On 05/01/2023 18:52, Vincent Carey wrote: > > > On Thu, Jan 5, 2023 at 7:08 PM Vincent Carey > <st...@channing.harvard.edu> wrote: > > > > On Thu, Jan 5, 2023 at 1:44 PM Hervé Pagès > <hpages.on.git...@gmail.com> wrote: > > Hi Martin, > > Linux runs on many architectures, ARM64 is just one of them. > > Our daily builds have traditionally focused on 3 platforms: > Intel-based > Linux (Ubuntu 22.04), Windows, and Intel-based Mac. Note that we > recently added ARM64-based Mac to our daily builds. > > One big difference between Linux and the other platforms is > that we only > produce binary packages for the latter. More precisely: > > - on the Linux builders: the daily builds only run 'R CMD > INSTALL', 'R > CMD build', and 'R CMD check', on each Bioconductor package, > > - on the Windows and Mac builders: the daily builds run all > the above > plus an additional step that we call the BUILD BIN step that > produces a > binary for each Bioconductor package. > > This means that on Linux, as well as on any other Unix-like OS > that is > not macOS (e.g. FreeBSD, OpenBSD, Solaris, HP-UX, etc...), > users will > install all their packages (Bioconductor and CRAN) **from > source**. This > should work as long as they are on a platform where R is > supported and > have the required compilers (C, C++, and Fortran). > > Note that if officially supporting a given platform means > running the > daily builds on that particular platform, then there's no way > for us to > do that because platform == OS + architecture, and the list of > combinations of Unix-like OS's (Linux, FreeBSD, Solaris, > etc...) + > architectures (Intel, ARM64, Sparc, powerpc) is endless. Even > if we > narrow this list to Intel-based Linux, there are hundreds of > Linux > distributions around that use different kernel, compilers, > package > managers, etc... > > All this to say that, as far as the daily builds are > concerned, we had > to make choices, and those choices are based on the most > commonly used > platforms. Since all Bioconductor packages are tested daily on > Intel-based Linux (Ubuntu 22.04), Windows, Intel-based Mac, and > ARM64-based Mac, we have some reasonable confidence that they > will work > properly on these 4 platforms (still not a 100% guarantee of > course, > there's nothing like that). > > My understanding is that ARM64-based Linux is still a > marginally used > platform so probably not worth for us to allocate resources on > adding it > to our daily builds at the moment. If it ever becomes more > mainstream in > the future, then we will certainly reconsider. That does not > mean that > you can't use Bioconductor on a ARM64-based Linux machine > **now**. I see > no reason a priori why you couldn't install (from source) > Bioconductor > packages on this platform, and use them, as long as: > > > Thanks Hervé for a good overview of the issues. I think there are > a couple > of reasons to keep this dialogue going (and there is now a > community slack channel > for further discussion: #arm-linux at community-bioc.slack.com > <http://community-bioc.slack.com>.) > > The first reason is Martin's offer of resources to accomplish the > support aim. What > exactly that support aim is remains to be made precise. As you > note, a properly > configured system with R can use BiocManager::install to build > from source, but > there are a few additional things that can be done to produce > binaries, and perhaps > some of our software in BBS or some of the binary repo generation > tools could be > useful for Martin's group to make a relevant binary repo. The > package-management > oriented process of Dirk Eddelbuettel's r2u > <https://github.com/eddelbuettel/r2u> also seems potentially > relevant. We also > have tooling to build all the CRAN dependencies that Bioc packages > declare. This > is all in the open and it would be interesting to see how much > work is needed to > get solutions for ARM64 linux. It could lead to some > robustification of the existing > build machinery. I am not offering to do it, but the fact that > all the tooling is out in > the open may not be fully clear and I am just mentioning this. > > The second reason to stay engaged is the nature of the ARM > platform, which is > said to require lower power consumption for equivalent > throughput. It may be > environmentally beneficial to be ahead of the curve in being able > to work with > this platform. Earlier I linked to a github issue indicating that > rocker now has a dual > platform container image including arm64 support but I don't know > if that really > addresses the issue at hand. Maybe I need to go onto a graviton > machine to find out. > > > So I did this, and here are some notes: > > 1) it is easy to get such a machine in AWS, a1.2xlarge > Linux 10a568f32a1c 4.14.296-222.539.amzn2.aarch64 #1 SMP Wed Oct 26 > 20:36:51 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux > 2) using the rocker/rstudio:latest-daily I could get DESeq2 installed > in about 20 minutes of > compilation of dependent packages > 3) to get a checkable version of DESeq2 I needed to enhance the > rocker environment > 4 apt-get install libxml2-dev > 8 apt install libpng-dev > 12 apt install libgit2-dev > 14 apt install -y libmagick++-dev > 16 apt install -y libharfbuzz-dev libfribidi-dev > 4) DESeq2 check in release version (1.38.2) failed (but it passes on > intel linux): > > Running examples in ‘DESeq2-Ex.R’ failed > The error most likely occurred in: > > > ### Name: unmix > > ### Title: Unmix samples using loss in a variance stabilized space > > ### Aliases: unmix > > > > ### ** Examples > > > > > > # some artificial data > > cts <- matrix(c(80,50,1,100, > + 1,1,60,100, > + 0,50,60,100), ncol=4, byrow=TRUE) > > # make a DESeqDataSet > > dds <- DESeqDataSetFromMatrix(cts, > + data.frame(row.names=seq_len(ncol(cts))), ~1) > converting counts to integer mode > > colnames(dds) <- paste0("sample",1:4) > > > > # note! here you would instead use > > # estimateSizeFactors() to do actual normalization > > sizeFactors(dds) <- rep(1, ncol(dds)) > > > > norm.cts <- counts(dds, normalized=TRUE) > > > > # 'pure' should also have normalized counts... > > pure <- matrix(c(10,0,0, > + 0,0,10, > + 0,10,0), ncol=3, byrow=TRUE) > > colnames(pure) <- letters[1:3] > > > > # for real data, you need to find alpha after fitting > estimateDispersions() > > mix <- unmix(norm.cts, pure, alpha=0.01) > Warning in sqrt(alpha * q) : NaNs produced > Error in optim(par = rep(1, ncol(pure)), fn = sumLossVST, gr = NULL, > i, : > L-BFGS-B needs finite values of 'fn' > Calls: unmix -> lapply -> lapply -> FUN -> optim Hmm.. this ain't good :-( > > Is there bugged/nonportable code somewhere in the stack underlying > this example?
Probably. > That could take some time to figure out. > > I conclude that the mechanics of working with ARM64 and R to process > Bioconductor > packages are very tractable, but the work needed to get the whole > ecosystem to a > favorable state, as usable as it is for intel linux or mac or windows, > may be laborious. OK so maybe a good start would be to try to set up the daily builds (BBS) on one of those AWS 1.2xlarge or a1.4xlarge instances, or, even better, on one of the VMs that Martin is offering? If we use ARM64 Ubuntu on it, setting up the builds there should be very similar to what we do for our current Intel Ubuntu build machines, which is easy and well-documented. H. > > > > > In any case it is not so often that we get a request for > enhancements that includes > an offer of VMs and person power so I want to be sure we don't > lose the thread > prematurely. > > > > > > - R is supported on your ARM64-based Linux machine > > - you have compilers that are supported by R > > - you have the external libraries that are required by some > CRAN and/or > Bioconductor packages. > > Hope this helps, > > H. > > On 05/01/2023 02:01, Martin Grigorov wrote: > > Dear community, > > > > Happy and successful new year! > > > > Appologies if this has been discussed before but > > https://stat.ethz.ch/pipermail/bioc-devel/ does not provide > search > > facilities and my googling didn't help much! > > > > I'd like to ask whether Linux ARM64 is officially supported ? > > I know that Mac ARM64 is supported since 3.16 [1] [2]. > > I cannot find such test results for Linux ARM64 and the site > search [3] > > also mentions "arm64" only in context of "macOS". > > In addition the Docker images are also single-platform [4] > (linux/amd64). > > > > How can we help to add support for Linux ARM64 ? > > My employer is willing to donate VMs and man power if the > community is > > interested in adding support for Linux ARM64! > > > > > > Regards, > > Martin > > > > 1. https://bioconductor.org/news/bioc_3_16_release/ > > 2. > https://bioconductor.org/checkResults/3.17/bioc-mac-arm64-LATEST/ > > 3. https://bioconductor.org/help/search/index.html?q=arm64/ > > 4. > https://hub.docker.com/r/bioconductor/bioconductor_docker/tags > > > > [[alternative HTML version deleted]] > > > > _______________________________________________ > > Bioc-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > -- > Hervé Pagès > > Bioconductor Core Team > hpages.on.git...@gmail.com > > _______________________________________________ > Bioc-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > The information in this e-mail is intended only for th...{{dropped:22}} _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
