Thanks for the suggestion. I am already only installing "nomkl" for macOS with "if (Sys.info()["sysname"] == "Darwin") "nomkl==3.0" else c()".
On Sat, Jul 8, 2023 at 11:09 PM Aaron Lun < infinite.monkeys.with.keyboa...@gmail.com> wrote: > Not hard to have OS-specific environments, see for example: > > https://github.com/alanocallaghan/snifter/blob/devel/R/basilisk.R > > -A > > On 7/6/23 20:23, Kasper Daniel Hansen wrote: > > This sounds excellent Kim! > > > > Here you can get 2.10 for Windows: > https://anaconda.org/anaconda/tensorflow > > although my experience is that I hate mixing channels on conda. It is > also > > quite interesting that this conda package also has Windows at an older > > version (but just 2.10 vs. 2.12) > > > > This really speaks to a potential need for having basilisk dependencies > > being platform specific. This would also come in handy for macOS. But > > AFAIK, this is not supported by basilisk currently. Might be something we > > need to address. > > > > Best, > > Kasper > > > > > > > > On Thu, Jul 6, 2023 at 5:38 PM Kim Philipp Jablonski < > > kim.philipp.jablon...@gmail.com> wrote: > > > >> Thank you all so much for your input and the references! > >> > >> @Kasper: I mostly rely on tensorflow and tensorflow-probability, so I > might > >> somehow get rid of the keras dependency but it would require some work. > >> > >> After being inspired by the lovely orthos package (thanks Charlotte!), I > >> decided to play around further in the basilisk direction and updated my > >> project (https://github.com/cbg-ethz/pareg/tree/feature-basilisk). > >> The hardest part was figuring out a set of package versions which > satisfy > >> conda's package manager (@Vincent, I feel you!). > >> But then it just magically worked on my local machine. > >> > >> When testing with GitHub Actions, the windows runner crashes with > >> PackagesNotFoundError: The following packages are not available from > >> current channels: > >> - tensorflow=2.11.1 > >> A look at conda-forge (https://anaconda.org/conda-forge/tensorflow) > >> reveals > >> that for win-64, there's only v1.14.0 available... I guess ignoring the > >> windows build is not an option for my bioc package? > >> > >> For the ubuntu runner, my vignettes were created successfully ("* > creating > >> vignettes ... OK"). My tests still fail, but that is expected because I > >> have not wrapped them with basiliskRun. Do I have to do this manually > for > >> every function call which may interact with tensorflow (so much > >> boilerplate), or can I somehow implicitly use the created conda env for > >> every function in my package? > >> > >> On Thu, Jul 6, 2023 at 2:08 PM Vincent Carey < > st...@channing.harvard.edu> > >> wrote: > >> > >>> That's great news. FWIW I am finding that the advice at > >>> https://rstudio.github.io/reticulate/articles/python_dependencies.html > >>> can work to produce properly resolved python dependencies. Just don't > >>> follow the example literally; the requested > >>> scipy version may not exist. Version 1.11.1 does. Stay tuned. > >>> > >>> On Thu, Jul 6, 2023 at 7:43 AM Charlotte Soneson < > >>> charlottesone...@gmail.com> > >>> wrote: > >>> > >>>> Hi, > >>>> > >>>> in case it's useful: we have a package (orthos) in review ( > >>>> https://github.com/Bioconductor/Contributions/issues/3042) which uses > >>>> basilisk to set up a conda environment with tensorflow and keras. It > >>> builds > >>>> and runs fine both on GitHub Actions (GitHub repo here: > >>>> https://github.com/fmicompbio/orthos) and on the Single Package > >> Builder. > >>>> We have also tested (locally) that it will use GPUs if available, and > >>> that > >>>> the GPU configuration can be controlled from the R session (outside of > >>> the > >>>> package), e.g. by setting the CUDA_VISIBLE_DEVICES environment > >> variable. > >>>> > >>>> Charlotte > >>>> > >>>>> On 5 Jul 2023, at 23:12, Kasper Daniel Hansen < > >>>> kasperdanielhan...@gmail.com> wrote: > >>>>> > >>>>> So I think Kim is interfacing to tensorflow by using the keras > >> package > >>>> from > >>>>> CRAN (partly authored by the Rstudio people). This package leaves it > >> to > >>>> the > >>>>> user to install tensorflow, which is a highly non-trivial > >> installation > >>>>> task. There is some partly helpful instructions for using conda > >>> together > >>>>> with reticulate (see the macOS tab on > >>>>> https://tensorflow.rstudio.com/install/local_gpu.html). This is the > >>> job > >>>>> that basilisk handles for you. In essence, basilisk allows the > >>> developer > >>>> to > >>>>> specify an R-package-specific conda. Tensorflow can be run on a > >> CPUor a > >>>>> GPU. Getting it to run on a user-GPU is extra complicated and I am > >> not > >>>> sure > >>>>> basilisk can handle this. > >>>>> > >>>>> Going forward, we (Bioc) want to decide if we want to support keras > >> on > >>>> our > >>>>> build system. This will require some work, because it is definitely > >> not > >>>>> trivial to get to work (but much more possible if we limit ourselves > >> to > >>>>> running on CPU). If we decide to support keras, we should try to > >> figure > >>>> out > >>>>> how to wrap keras into a basilisk container; perhaps something like > >>>>> creating a keras-basilisk R package, because IF we decide to support > >>>> keras, > >>>>> this is going to be a major headache (to add to the frustration, > >>>> tensorflow > >>>>> often rearranges everything so I future issues keeping it > >> operational). > >>>>> > >>>>> For Kim: I think you should consider if there are any alternatives to > >>>>> keras. Even if we get it to work on our build system, users will have > >>>> major > >>>>> headache getting this to work (I think). If there are no alternatives > >>> to > >>>>> keras, you should perhaps think about doing the keras-basilisk > >> option I > >>>>> outline above (assuming that is feasible; I don't know how keras > >>>> interfaces > >>>>> with tensorflow. You might also have major headaches in your future: > >> I > >>>> have > >>>>> found quite big differences in convergence and optimizers over time > >> in > >>> TF > >>>>> and you're basically saying it should work with every version of > >>>> tensorflow > >>>>>> = 2.2. That's a .... strong promise considering also the difference > >>>>> between GPU and CPU. > >>>>> > >>>>> In the meantime, in case the core build team wants some input, I have > >>>> some > >>>>> experience with tensorflow, although so far my experience is mostly > >>>>> frustration. > >>>>> > >>>>> Best, > >>>>> Kasper > >>>>> > >>>>> > >>>>> On Wed, Jul 5, 2023 at 3:12 PM Vincent Carey < > >>> st...@channing.harvard.edu > >>>>> > >>>>> wrote: > >>>>> > >>>>>> I'll try to get clearer on the basilisk situation; I forked your > >> repo > >>>>>> and will plunge in soon. > >>>>>> It may take a while. In the mean time I hope the BBS python stack > >> can > >>>>>> be looked at > >>>>>> to see what the issue might be. @Jennifer Wokaty is it possible > >> with > >>>>>> the recent R upgrades that > >>>>>> tensorflow and tensorflow-probability might need to be > >>>> installed/updated? > >>>>>> > >>>>>> On Wed, Jul 5, 2023 at 3:01 PM Kim Philipp Jablonski > >>>>>> <kim.philipp.jablon...@gmail.com> wrote: > >>>>>>> > >>>>>>> Thanks a lot for your response! > >>>>>>> > >>>>>>> The Config/reticulate approach in DESCRIPTION looks very neat. I > >> hope > >>>> we > >>>>>> can use it at some point. > >>>>>>> > >>>>>>> Could the issue you are facing with the conda install be, that the > >>>>>> package is called "tensorflow-probability" instead of > >>>>>> "tensorflow_probability" ( > >>>>>> https://anaconda.org/conda-forge/tensorflow-probability)? > >>>>>>> > >>>>>>> I have followed the basilisk advice and incorporated it into my > >>> package > >>>>>> here: https://github.com/cbg-ethz/pareg/tree/feature-basilisk. > >>>>>>> The basilisk docs say "Any R functions that use Python code should > >> do > >>>> so > >>>>>> via basiliskRun()". > >>>>>>> This does not seem possible in my case as I am using > >>>>>> tensorflow(probability) wrappers instead of calling them directly. > >>>>>>> Building the vignettes with Github Actions fails because tensorflow > >>>>>> could not be found: > >>>>>> > >>>> > >>> > >> > https://github.com/cbg-ethz/pareg/actions/runs/5466980003/jobs/9952690137#step:15:39 > >>>>>> (I might have to activate basilisk somehow, but using basiliskStart > >>>> seems > >>>>>> to only make sense with basiliskRun). I could also not find any > >> advice > >>>> on > >>>>>> this elsewhere. > >>>>>>> Do you have a suggestion how to best handle this (or should it > >> *just* > >>>>>> work)? > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Jul 4, 2023 at 1:04 PM Vincent Carey < > >>>> st...@channing.harvard.edu> > >>>>>> wrote: > >>>>>>>> > >>>>>>>> Sorry to hear about this. Our informal outlook on python > >>>>>>>> interoperation in Bioconductor packages is that > >>>>>>>> the basilisk discipline should be used. This allows you to pin > >>>>>>>> specific versions of all python dependencies > >>>>>>>> and use python for your R functions from an insulated conda > >>>> environment. > >>>>>>>> > >>>>>>>> There is also information about specifying python dependencies in > >>> the > >>>>>>>> DESCRIPTION file at > >>>>>>>> > >>>> > https://rstudio.github.io/reticulate/articles/python_dependencies.html > >>>>>>>> I do not know if our build > >>>>>>>> system or R CMD install take effective advantage of that approach > >> at > >>>>>> this time. > >>>>>>>> > >>>>>>>> I was surprised to see that my attempt to use > >>> reticulate::py_install() > >>>>>>>> to install tensorflow_probabiliity > >>>>>>>> on my laptop failed: > >>>>>>>> > >>>>>>>> '/home/stvjc/.local/share/r-miniconda/bin/conda' 'install' '--yes' > >>>>>>>> '--prefix' > >> '/home/stvjc/.local/share/r-miniconda/envs/r-reticulate' > >>>>>>>> '-c' 'conda-forge' 'tensorflow_probability' > >>>>>>>> Retrieving notices: ...working... done > >>>>>>>> Collecting package metadata (current_repodata.json): done > >>>>>>>> Solving environment: unsuccessful initial attempt using frozen > >>> solve. > >>>>>>>> Retrying with flexible solve. > >>>>>>>> Collecting package metadata (repodata.json): done > >>>>>>>> Solving environment: unsuccessful initial attempt using frozen > >>> solve. > >>>>>>>> Retrying with flexible solve. > >>>>>>>> > >>>>>>>> PackagesNotFoundError: The following packages are not available > >> from > >>>>>>>> current channels: > >>>>>>>> > >>>>>>>> - tensorflow_probability > >>>>>>>> > >>>>>>>> Current channels: > >>>>>>>> > >>>>>>>> - https://conda.anaconda.org/conda-forge/linux-64 > >>>>>>>> - https://conda.anaconda.org/conda-forge/noarch > >>>>>>>> - https://repo.anaconda.com/pkgs/main/linux-64 > >>>>>>>> - https://repo.anaconda.com/pkgs/main/noarch > >>>>>>>> - https://repo.anaconda.com/pkgs/r/linux-64 > >>>>>>>> - https://repo.anaconda.com/pkgs/r/noarch > >>>>>>>> > >>>>>>>> To search for alternate channels that may provide the conda > >> package > >>>>>> you're > >>>>>>>> looking for, navigate to > >>>>>>>> > >>>>>>>> https://anaconda.org > >>>>>>>> > >>>>>>>> and use the search bar at the top of the page. > >>>>>>>> > >>>>>>>> > >>>>>>>> Error: one or more Python packages failed to install [error code > >> 1] > >>>>>>>> > >>>>>>>> Enter a frame number, or 0 to exit > >>>>>>>> > >>>>>>>> 1: py_install("tensorflow_probability") > >>>>>>>> > >>>>>>>> So the "current channels" used by reticulate in my pretty stock > >>>>>>>> installation of R/reticulate seem flawed > >>>>>>>> for this purpose. I had hoped to write a bit of code that would > >>> check > >>>>>>>> for the desired module and install > >>>>>>>> it if missing, that you could include in your package. > >>>>>>>> > >>>>>>>> get_pippath = function() { > >>>>>>>> pypath = reticulate::conda_list() |> > >>>>>> (\(x)x[x$name=="r-reticulate",])() > >>>>>>>> gsub("python$", "pip", pypath$python) > >>>>>>>> } > >>>>>>>> > >>>>>>>> check_tfp = function() { > >>>>>>>> pippath = get_pippath() > >>>>>>>> peek = system(paste0(pippath, " list | grep tensor"), > >> intern=TRUE) > >>>>>>>> peek = gsub(" +", " ", peek) > >>>>>>>> tfdf = do.call(rbind, lapply(strsplit(peek, " "), function(x) > >>>>>>>> data.frame(pkg=x[1], version=x[2]))) > >>>>>>>> "tensorflow_probability" %in% tfdf$pkg > >>>>>>>> } > >>>>>>>> > >>>>>>>> install_tfp = function() { > >>>>>>>> pippath = get_pippath() > >>>>>>>> chk = system(paste(pippath, "install tensorflow_probability"), > >>>>>> intern=TRUE) > >>>>>>>> chk2 = check_tfp() > >>>>>>>> list(sysout=chk, tfp_installed=chk2) > >>>>>>>> } > >>>>>>>> > >>>>>>>> The first two functions seem to do what I want, the latter does > >> not. > >>>>>>>> > >>>>>>>> Suggestion -- read the basilisk vignettes, use it if at all > >>> possible. > >>>>>>>> Certainly we can take > >>>>>>>> care of this in our build system when the holiday ends, but it > >> would > >>>>>>>> be great to have the software > >>>>>>>> arrange the solution when necessary, and I don't see a way to > >>>>>>>> accomplish this at the moment. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, Jul 4, 2023 at 5:32 AM Kim Philipp Jablonski > >>>>>>>> <kim.philipp.jablon...@gmail.com> wrote: > >>>>>>>>> > >>>>>>>>> Hello, > >>>>>>>>> > >>>>>>>>> The latest builds of my package 'pareg' started failing on BioC > >>> 3.17 > >>>>>> for > >>>>>>>>> Linux and macOS but not Windows: > >>>>>>>>> > >>>>>> > >>>> > >>> > >> > https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/pareg/nebbiolo1-buildsrc.html > >>>>>>>>> . > >>>>>>>>> In both cases, the error messages is "Python module > >>>>>> tensorflow_probability > >>>>>>>>> was not found". This wasn't an issue in earlier releases. > >>>>>>>>> > >>>>>>>>> Do you know what the reason for this is and how I can fix it? > >>>>>>>>> Thanks a lot for your help! > >>>>>>>>> > >>>>>>>>> Best regards, > >>>>>>>>> Kim > >>>>>>>>> > >>>>>>>>> [[alternative HTML version deleted]] > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> Bioc-devel@r-project.org mailing list > >>>>>>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel > >>>>>>>> > >>>>>>>> -- > >>>>>>>> The information in this e-mail is intended only for the person to > >>> whom > >>>>>> it > >>>>>>>> is > >>>>>>>> addressed. If you believe this e-mail was sent to you in error and > >>> the > >>>>>>>> e-mail > >>>>>>>> contains patient information, please contact the Partners > >> Compliance > >>>>>>>> HelpLine at > >>>>>>>> http://www.partners.org/complianceline > >>>>>>>> <http://www.partners.org/complianceline> . If the e-mail was sent > >>> to > >>>>>> you in > >>>>>>>> error > >>>>>>>> but does not contain patient information, please contact the > >> sender > >>>>>>>> and properly > >>>>>>>> dispose of the e-mail. > >>>>>> > >>>>>> -- > >>>>>> The information in this e-mail is intended only for > >>> th...{{dropped:27}} > >>>>> > >>>>> _______________________________________________ > >>>>> Bioc-devel@r-project.org mailing list > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel > >>>> > >>>> > >>> > >>> -- > >>> The information in this e-mail is intended only for th...{{dropped:10}} > >> > >> _______________________________________________ > >> Bioc-devel@r-project.org mailing list > >> https://stat.ethz.ch/mailman/listinfo/bioc-devel > >> > > > > > > _______________________________________________ > Bioc-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel > [[alternative HTML version deleted]] _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
