Not hard to have OS-specific environments, see for example:
https://github.com/alanocallaghan/snifter/blob/devel/R/basilisk.R
-A
On 7/6/23 20:23, Kasper Daniel Hansen wrote:
This sounds excellent Kim!
Here you can get 2.10 for Windows: https://anaconda.org/anaconda/tensorflow
although my experience is that I hate mixing channels on conda. It is also
quite interesting that this conda package also has Windows at an older
version (but just 2.10 vs. 2.12)
This really speaks to a potential need for having basilisk dependencies
being platform specific. This would also come in handy for macOS. But
AFAIK, this is not supported by basilisk currently. Might be something we
need to address.
Best,
Kasper
On Thu, Jul 6, 2023 at 5:38 PM Kim Philipp Jablonski <
kim.philipp.jablon...@gmail.com> wrote:
Thank you all so much for your input and the references!
@Kasper: I mostly rely on tensorflow and tensorflow-probability, so I might
somehow get rid of the keras dependency but it would require some work.
After being inspired by the lovely orthos package (thanks Charlotte!), I
decided to play around further in the basilisk direction and updated my
project (https://github.com/cbg-ethz/pareg/tree/feature-basilisk).
The hardest part was figuring out a set of package versions which satisfy
conda's package manager (@Vincent, I feel you!).
But then it just magically worked on my local machine.
When testing with GitHub Actions, the windows runner crashes with
PackagesNotFoundError: The following packages are not available from
current channels:
- tensorflow=2.11.1
A look at conda-forge (https://anaconda.org/conda-forge/tensorflow)
reveals
that for win-64, there's only v1.14.0 available... I guess ignoring the
windows build is not an option for my bioc package?
For the ubuntu runner, my vignettes were created successfully ("* creating
vignettes ... OK"). My tests still fail, but that is expected because I
have not wrapped them with basiliskRun. Do I have to do this manually for
every function call which may interact with tensorflow (so much
boilerplate), or can I somehow implicitly use the created conda env for
every function in my package?
On Thu, Jul 6, 2023 at 2:08 PM Vincent Carey <st...@channing.harvard.edu>
wrote:
That's great news. FWIW I am finding that the advice at
https://rstudio.github.io/reticulate/articles/python_dependencies.html
can work to produce properly resolved python dependencies. Just don't
follow the example literally; the requested
scipy version may not exist. Version 1.11.1 does. Stay tuned.
On Thu, Jul 6, 2023 at 7:43 AM Charlotte Soneson <
charlottesone...@gmail.com>
wrote:
Hi,
in case it's useful: we have a package (orthos) in review (
https://github.com/Bioconductor/Contributions/issues/3042) which uses
basilisk to set up a conda environment with tensorflow and keras. It
builds
and runs fine both on GitHub Actions (GitHub repo here:
https://github.com/fmicompbio/orthos) and on the Single Package
Builder.
We have also tested (locally) that it will use GPUs if available, and
that
the GPU configuration can be controlled from the R session (outside of
the
package), e.g. by setting the CUDA_VISIBLE_DEVICES environment
variable.
Charlotte
On 5 Jul 2023, at 23:12, Kasper Daniel Hansen <
kasperdanielhan...@gmail.com> wrote:
So I think Kim is interfacing to tensorflow by using the keras
package
from
CRAN (partly authored by the Rstudio people). This package leaves it
to
the
user to install tensorflow, which is a highly non-trivial
installation
task. There is some partly helpful instructions for using conda
together
with reticulate (see the macOS tab on
https://tensorflow.rstudio.com/install/local_gpu.html). This is the
job
that basilisk handles for you. In essence, basilisk allows the
developer
to
specify an R-package-specific conda. Tensorflow can be run on a
CPUor a
GPU. Getting it to run on a user-GPU is extra complicated and I am
not
sure
basilisk can handle this.
Going forward, we (Bioc) want to decide if we want to support keras
on
our
build system. This will require some work, because it is definitely
not
trivial to get to work (but much more possible if we limit ourselves
to
running on CPU). If we decide to support keras, we should try to
figure
out
how to wrap keras into a basilisk container; perhaps something like
creating a keras-basilisk R package, because IF we decide to support
keras,
this is going to be a major headache (to add to the frustration,
tensorflow
often rearranges everything so I future issues keeping it
operational).
For Kim: I think you should consider if there are any alternatives to
keras. Even if we get it to work on our build system, users will have
major
headache getting this to work (I think). If there are no alternatives
to
keras, you should perhaps think about doing the keras-basilisk
option I
outline above (assuming that is feasible; I don't know how keras
interfaces
with tensorflow. You might also have major headaches in your future:
I
have
found quite big differences in convergence and optimizers over time
in
TF
and you're basically saying it should work with every version of
tensorflow
= 2.2. That's a .... strong promise considering also the difference
between GPU and CPU.
In the meantime, in case the core build team wants some input, I have
some
experience with tensorflow, although so far my experience is mostly
frustration.
Best,
Kasper
On Wed, Jul 5, 2023 at 3:12 PM Vincent Carey <
st...@channing.harvard.edu
wrote:
I'll try to get clearer on the basilisk situation; I forked your
repo
and will plunge in soon.
It may take a while. In the mean time I hope the BBS python stack
can
be looked at
to see what the issue might be. @Jennifer Wokaty is it possible
with
the recent R upgrades that
tensorflow and tensorflow-probability might need to be
installed/updated?
On Wed, Jul 5, 2023 at 3:01 PM Kim Philipp Jablonski
<kim.philipp.jablon...@gmail.com> wrote:
Thanks a lot for your response!
The Config/reticulate approach in DESCRIPTION looks very neat. I
hope
we
can use it at some point.
Could the issue you are facing with the conda install be, that the
package is called "tensorflow-probability" instead of
"tensorflow_probability" (
https://anaconda.org/conda-forge/tensorflow-probability)?
I have followed the basilisk advice and incorporated it into my
package
here: https://github.com/cbg-ethz/pareg/tree/feature-basilisk.
The basilisk docs say "Any R functions that use Python code should
do
so
via basiliskRun()".
This does not seem possible in my case as I am using
tensorflow(probability) wrappers instead of calling them directly.
Building the vignettes with Github Actions fails because tensorflow
could not be found:
https://github.com/cbg-ethz/pareg/actions/runs/5466980003/jobs/9952690137#step:15:39
(I might have to activate basilisk somehow, but using basiliskStart
seems
to only make sense with basiliskRun). I could also not find any
advice
on
this elsewhere.
Do you have a suggestion how to best handle this (or should it
*just*
work)?
On Tue, Jul 4, 2023 at 1:04 PM Vincent Carey <
st...@channing.harvard.edu>
wrote:
Sorry to hear about this. Our informal outlook on python
interoperation in Bioconductor packages is that
the basilisk discipline should be used. This allows you to pin
specific versions of all python dependencies
and use python for your R functions from an insulated conda
environment.
There is also information about specifying python dependencies in
the
DESCRIPTION file at
https://rstudio.github.io/reticulate/articles/python_dependencies.html
I do not know if our build
system or R CMD install take effective advantage of that approach
at
this time.
I was surprised to see that my attempt to use
reticulate::py_install()
to install tensorflow_probabiliity
on my laptop failed:
'/home/stvjc/.local/share/r-miniconda/bin/conda' 'install' '--yes'
'--prefix'
'/home/stvjc/.local/share/r-miniconda/envs/r-reticulate'
'-c' 'conda-forge' 'tensorflow_probability'
Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen
solve.
Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen
solve.
Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available
from
current channels:
- tensorflow_probability
Current channels:
- https://conda.anaconda.org/conda-forge/linux-64
- https://conda.anaconda.org/conda-forge/noarch
- https://repo.anaconda.com/pkgs/main/linux-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/linux-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda
package
you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Error: one or more Python packages failed to install [error code
1]
Enter a frame number, or 0 to exit
1: py_install("tensorflow_probability")
So the "current channels" used by reticulate in my pretty stock
installation of R/reticulate seem flawed
for this purpose. I had hoped to write a bit of code that would
check
for the desired module and install
it if missing, that you could include in your package.
get_pippath = function() {
pypath = reticulate::conda_list() |>
(\(x)x[x$name=="r-reticulate",])()
gsub("python$", "pip", pypath$python)
}
check_tfp = function() {
pippath = get_pippath()
peek = system(paste0(pippath, " list | grep tensor"),
intern=TRUE)
peek = gsub(" +", " ", peek)
tfdf = do.call(rbind, lapply(strsplit(peek, " "), function(x)
data.frame(pkg=x[1], version=x[2])))
"tensorflow_probability" %in% tfdf$pkg
}
install_tfp = function() {
pippath = get_pippath()
chk = system(paste(pippath, "install tensorflow_probability"),
intern=TRUE)
chk2 = check_tfp()
list(sysout=chk, tfp_installed=chk2)
}
The first two functions seem to do what I want, the latter does
not.
Suggestion -- read the basilisk vignettes, use it if at all
possible.
Certainly we can take
care of this in our build system when the holiday ends, but it
would
be great to have the software
arrange the solution when necessary, and I don't see a way to
accomplish this at the moment.
On Tue, Jul 4, 2023 at 5:32 AM Kim Philipp Jablonski
<kim.philipp.jablon...@gmail.com> wrote:
Hello,
The latest builds of my package 'pareg' started failing on BioC
3.17
for
Linux and macOS but not Windows:
https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/pareg/nebbiolo1-buildsrc.html
.
In both cases, the error messages is "Python module
tensorflow_probability
was not found". This wasn't an issue in earlier releases.
Do you know what the reason for this is and how I can fix it?
Thanks a lot for your help!
Best regards,
Kim
[[alternative HTML version deleted]]
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
--
The information in this e-mail is intended only for the person to
whom
it
is
addressed. If you believe this e-mail was sent to you in error and
the
e-mail
contains patient information, please contact the Partners
Compliance
HelpLine at
http://www.partners.org/complianceline
<http://www.partners.org/complianceline> . If the e-mail was sent
to
you in
error
but does not contain patient information, please contact the
sender
and properly
dispose of the e-mail.
--
The information in this e-mail is intended only for
th...{{dropped:27}}
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
--
The information in this e-mail is intended only for th...{{dropped:10}}
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
