That's great news. FWIW I am finding that the advice at https://rstudio.github.io/reticulate/articles/python_dependencies.html can work to produce properly resolved python dependencies. Just don't follow the example literally; the requested scipy version may not exist. Version 1.11.1 does. Stay tuned.
On Thu, Jul 6, 2023 at 7:43 AM Charlotte Soneson <charlottesone...@gmail.com> wrote: > Hi, > > in case it's useful: we have a package (orthos) in review ( > https://github.com/Bioconductor/Contributions/issues/3042) which uses > basilisk to set up a conda environment with tensorflow and keras. It builds > and runs fine both on GitHub Actions (GitHub repo here: > https://github.com/fmicompbio/orthos) and on the Single Package Builder. > We have also tested (locally) that it will use GPUs if available, and that > the GPU configuration can be controlled from the R session (outside of the > package), e.g. by setting the CUDA_VISIBLE_DEVICES environment variable. > > Charlotte > > > On 5 Jul 2023, at 23:12, Kasper Daniel Hansen < > kasperdanielhan...@gmail.com> wrote: > > > > So I think Kim is interfacing to tensorflow by using the keras package > from > > CRAN (partly authored by the Rstudio people). This package leaves it to > the > > user to install tensorflow, which is a highly non-trivial installation > > task. There is some partly helpful instructions for using conda together > > with reticulate (see the macOS tab on > > https://tensorflow.rstudio.com/install/local_gpu.html). This is the job > > that basilisk handles for you. In essence, basilisk allows the developer > to > > specify an R-package-specific conda. Tensorflow can be run on a CPUor a > > GPU. Getting it to run on a user-GPU is extra complicated and I am not > sure > > basilisk can handle this. > > > > Going forward, we (Bioc) want to decide if we want to support keras on > our > > build system. This will require some work, because it is definitely not > > trivial to get to work (but much more possible if we limit ourselves to > > running on CPU). If we decide to support keras, we should try to figure > out > > how to wrap keras into a basilisk container; perhaps something like > > creating a keras-basilisk R package, because IF we decide to support > keras, > > this is going to be a major headache (to add to the frustration, > tensorflow > > often rearranges everything so I future issues keeping it operational). > > > > For Kim: I think you should consider if there are any alternatives to > > keras. Even if we get it to work on our build system, users will have > major > > headache getting this to work (I think). If there are no alternatives to > > keras, you should perhaps think about doing the keras-basilisk option I > > outline above (assuming that is feasible; I don't know how keras > interfaces > > with tensorflow. You might also have major headaches in your future: I > have > > found quite big differences in convergence and optimizers over time in TF > > and you're basically saying it should work with every version of > tensorflow > >> = 2.2. That's a .... strong promise considering also the difference > > between GPU and CPU. > > > > In the meantime, in case the core build team wants some input, I have > some > > experience with tensorflow, although so far my experience is mostly > > frustration. > > > > Best, > > Kasper > > > > > > On Wed, Jul 5, 2023 at 3:12 PM Vincent Carey <st...@channing.harvard.edu > > > > wrote: > > > >> I'll try to get clearer on the basilisk situation; I forked your repo > >> and will plunge in soon. > >> It may take a while. In the mean time I hope the BBS python stack can > >> be looked at > >> to see what the issue might be. @Jennifer Wokaty is it possible with > >> the recent R upgrades that > >> tensorflow and tensorflow-probability might need to be > installed/updated? > >> > >> On Wed, Jul 5, 2023 at 3:01 PM Kim Philipp Jablonski > >> <kim.philipp.jablon...@gmail.com> wrote: > >>> > >>> Thanks a lot for your response! > >>> > >>> The Config/reticulate approach in DESCRIPTION looks very neat. I hope > we > >> can use it at some point. > >>> > >>> Could the issue you are facing with the conda install be, that the > >> package is called "tensorflow-probability" instead of > >> "tensorflow_probability" ( > >> https://anaconda.org/conda-forge/tensorflow-probability)? > >>> > >>> I have followed the basilisk advice and incorporated it into my package > >> here: https://github.com/cbg-ethz/pareg/tree/feature-basilisk. > >>> The basilisk docs say "Any R functions that use Python code should do > so > >> via basiliskRun()". > >>> This does not seem possible in my case as I am using > >> tensorflow(probability) wrappers instead of calling them directly. > >>> Building the vignettes with Github Actions fails because tensorflow > >> could not be found: > >> > https://github.com/cbg-ethz/pareg/actions/runs/5466980003/jobs/9952690137#step:15:39 > >> (I might have to activate basilisk somehow, but using basiliskStart > seems > >> to only make sense with basiliskRun). I could also not find any advice > on > >> this elsewhere. > >>> Do you have a suggestion how to best handle this (or should it *just* > >> work)? > >>> > >>> > >>> On Tue, Jul 4, 2023 at 1:04 PM Vincent Carey < > st...@channing.harvard.edu> > >> wrote: > >>>> > >>>> Sorry to hear about this. Our informal outlook on python > >>>> interoperation in Bioconductor packages is that > >>>> the basilisk discipline should be used. This allows you to pin > >>>> specific versions of all python dependencies > >>>> and use python for your R functions from an insulated conda > environment. > >>>> > >>>> There is also information about specifying python dependencies in the > >>>> DESCRIPTION file at > >>>> > https://rstudio.github.io/reticulate/articles/python_dependencies.html > >>>> I do not know if our build > >>>> system or R CMD install take effective advantage of that approach at > >> this time. > >>>> > >>>> I was surprised to see that my attempt to use reticulate::py_install() > >>>> to install tensorflow_probabiliity > >>>> on my laptop failed: > >>>> > >>>> '/home/stvjc/.local/share/r-miniconda/bin/conda' 'install' '--yes' > >>>> '--prefix' '/home/stvjc/.local/share/r-miniconda/envs/r-reticulate' > >>>> '-c' 'conda-forge' 'tensorflow_probability' > >>>> Retrieving notices: ...working... done > >>>> Collecting package metadata (current_repodata.json): done > >>>> Solving environment: unsuccessful initial attempt using frozen solve. > >>>> Retrying with flexible solve. > >>>> Collecting package metadata (repodata.json): done > >>>> Solving environment: unsuccessful initial attempt using frozen solve. > >>>> Retrying with flexible solve. > >>>> > >>>> PackagesNotFoundError: The following packages are not available from > >>>> current channels: > >>>> > >>>> - tensorflow_probability > >>>> > >>>> Current channels: > >>>> > >>>> - https://conda.anaconda.org/conda-forge/linux-64 > >>>> - https://conda.anaconda.org/conda-forge/noarch > >>>> - https://repo.anaconda.com/pkgs/main/linux-64 > >>>> - https://repo.anaconda.com/pkgs/main/noarch > >>>> - https://repo.anaconda.com/pkgs/r/linux-64 > >>>> - https://repo.anaconda.com/pkgs/r/noarch > >>>> > >>>> To search for alternate channels that may provide the conda package > >> you're > >>>> looking for, navigate to > >>>> > >>>> https://anaconda.org > >>>> > >>>> and use the search bar at the top of the page. > >>>> > >>>> > >>>> Error: one or more Python packages failed to install [error code 1] > >>>> > >>>> Enter a frame number, or 0 to exit > >>>> > >>>> 1: py_install("tensorflow_probability") > >>>> > >>>> So the "current channels" used by reticulate in my pretty stock > >>>> installation of R/reticulate seem flawed > >>>> for this purpose. I had hoped to write a bit of code that would check > >>>> for the desired module and install > >>>> it if missing, that you could include in your package. > >>>> > >>>> get_pippath = function() { > >>>> pypath = reticulate::conda_list() |> > >> (\(x)x[x$name=="r-reticulate",])() > >>>> gsub("python$", "pip", pypath$python) > >>>> } > >>>> > >>>> check_tfp = function() { > >>>> pippath = get_pippath() > >>>> peek = system(paste0(pippath, " list | grep tensor"), intern=TRUE) > >>>> peek = gsub(" +", " ", peek) > >>>> tfdf = do.call(rbind, lapply(strsplit(peek, " "), function(x) > >>>> data.frame(pkg=x[1], version=x[2]))) > >>>> "tensorflow_probability" %in% tfdf$pkg > >>>> } > >>>> > >>>> install_tfp = function() { > >>>> pippath = get_pippath() > >>>> chk = system(paste(pippath, "install tensorflow_probability"), > >> intern=TRUE) > >>>> chk2 = check_tfp() > >>>> list(sysout=chk, tfp_installed=chk2) > >>>> } > >>>> > >>>> The first two functions seem to do what I want, the latter does not. > >>>> > >>>> Suggestion -- read the basilisk vignettes, use it if at all possible. > >>>> Certainly we can take > >>>> care of this in our build system when the holiday ends, but it would > >>>> be great to have the software > >>>> arrange the solution when necessary, and I don't see a way to > >>>> accomplish this at the moment. > >>>> > >>>> > >>>> > >>>> On Tue, Jul 4, 2023 at 5:32 AM Kim Philipp Jablonski > >>>> <kim.philipp.jablon...@gmail.com> wrote: > >>>>> > >>>>> Hello, > >>>>> > >>>>> The latest builds of my package 'pareg' started failing on BioC 3.17 > >> for > >>>>> Linux and macOS but not Windows: > >>>>> > >> > https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/pareg/nebbiolo1-buildsrc.html > >>>>> . > >>>>> In both cases, the error messages is "Python module > >> tensorflow_probability > >>>>> was not found". This wasn't an issue in earlier releases. > >>>>> > >>>>> Do you know what the reason for this is and how I can fix it? > >>>>> Thanks a lot for your help! > >>>>> > >>>>> Best regards, > >>>>> Kim > >>>>> > >>>>> [[alternative HTML version deleted]] > >>>>> > >>>>> _______________________________________________ > >>>>> Bioc-devel@r-project.org mailing list > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel > >>>> > >>>> -- > >>>> The information in this e-mail is intended only for the person to whom > >> it > >>>> is > >>>> addressed. If you believe this e-mail was sent to you in error and the > >>>> e-mail > >>>> contains patient information, please contact the Partners Compliance > >>>> HelpLine at > >>>> http://www.partners.org/complianceline > >>>> <http://www.partners.org/complianceline> . If the e-mail was sent to > >> you in > >>>> error > >>>> but does not contain patient information, please contact the sender > >>>> and properly > >>>> dispose of the e-mail. > >> > >> -- > >> The information in this e-mail is intended only for th...{{dropped:27}} > > > > _______________________________________________ > > Bioc-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > -- The information in this e-mail is intended only for the ...{{dropped:18}} _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
