Hi. Keeping in mind I am very new to easybuild, I would like to clarify a couple of points.
The script I linked to, works off an installation of R that already has the required package installed, so it would not be able to help you in creating a easyconfig "from scratch". The aim of easybuild (for what I can say) is to specify a detailed, tested and immutable set of packages so that reproducibility can be archived. I use the script I provided as a tool to help me come up with that list. Here how I would work: - install a vanilla R (with easybuild or otherwise) - install required packages with install.packages (or otherwise) - test - use the script provided to take a snapshot of the installation and create a easyconfig with list of packages in the right order and version. - deploy somewhere else or create a reproducible build I think that when you want to install ggplot2, there are several combinations of packages that would do, even if you set a particular version of ggplot2. CRAN will give you the latest version for each package, so it is not very stable or predictable. you can also look into packrat (http://cran.r-project.org/web/packages/packrat/index.html) for archiving reproducibility. Maybe you can borrow some ideas there too. I hope this help. Stefano > -----Original Message----- > From: [email protected] [mailto:easybuild- > [email protected]] On Behalf Of Pieter Neerincx > Sent: 24 June 2015 11:54 > To: [email protected] > Subject: Re: [easybuild] Easier way of installing R packages > > Hi Kenneth, > > On Jun 24, 2015, at 11:10 AM, Kenneth Hoste <[email protected]> > wrote: > > > Hi Pieter, > > > > On 24/06/15 10:58, Pieter Neerincx wrote: > >> Hi, > >> > >> On Jun 24, 2015, at 7:34 AM, Kenneth Hoste <[email protected]> > wrote: > >> > >>> Hi Riccardo, > >>> > >>> On 19/06/15 10:32, Riccardo Murri wrote: > >>>> Hi Ken, all, > >>>> > >>>> (Kenneth Hoste, Thu, Jun 18, 2015 at 04:32:27PM +0200:) > >>>>> We deliberately do not enable 'dependency=True', to ensure we can > reproduce > >>>>> the exact same installation later; if you let R automagically resolve > >>>>> dependencies and let it pick the most recent version at that time for > the > >>>>> dependencies, you can't do an exact reproduced install. > >>>> Could this be worked around by having EB save the list of installed R > >>>> packages + versions (like `pip freeze`) upon first install, then re-use > >>>> the saved list if available. > >>>> > >>>> Something along these lines (pseudo-code): > >>>> > >>>> package_list_file = easyconfig_file_name.replace('.eb', > '.pkglist.csv') > >>>> if not exists(package_list_file): > >>>> # 1. install packages by name, let R choose version > >>>> # 2. dump list of installed packages to file: (pkgname, > >>>> version) > >>>> else: > >>>> # 1. read in the pkglist.csv file > >>>> # 2. use R's install_version() to install the exact same pkg > >>>> and > version > >>> I like this idea... > >> +1 > >> > >>> We could have an easyconfig parameter like 'auto_deps = True', and > when that is set, EasyBuild lets R resolve dependencies by itself. > >>> > >>> Once the full list of easyconfigs has been installed, EasyBuild could then > see what got installed, and which versions, and adjust the list of extensions > included in the easyconfig file. The tricky bit may be the order, but there's > probably a solution for that (see Stefano's R script). > >>> > >>> Now, who's up for implementing this? :-) > >> Yesterday I started working on an R-script that generates an *.eb from an > existing R installation. This is based on Stefano's code (Many thanks for > sharing that!) and adds some commandline switches, help and code to figure > out whether a package came from a CRAN repo or from BioConductor. Hence > this is slightly different compared to the above as it generates a complete > *.eb from pure R, but I can easily add an option to only generate a pkglist > file, > which may then be used by an existing *.eb. > >> > >> Is it Ok if I add this with a README to the ..../r/R/ dir in the > >> EasyConfigs > repo and create a pull request or would you prefer this kind of > supplementary code to go elsewhere? > > > > It would be better if you could integrate this in the R easyblock somehow (I > realize that may be a lot to ask for). > > The small amount of code that generates the *.eb could be in Python and > hence that could go in the easyblock, but the majority of the code is R based > on Stefano's script to generate the list of R packages their versions & the > repos they came from and then re-order the list based on the dependencies. > I would not know how to accomplish this without using R's own dependency > handling mechanism and AFAIK their is no way to do this in Python. > > So the best I can imagine for now is a modified R easyblock that can execute > an external R script to create a pkglist... and then generate an updated > easyconfig. > > > We currently don't have a good location for this; I don't think we should > start adding 'random' scripts to the easyconfigs repo... > > The .../r/R/ dir in the EasyConfigs repo would be the first place where I > would > look for examples and a way to create new updated R easyconfigs, so in that > sense it isn't a random location, but I agree that it is random in the sense > that > it isn't EasyBuild style Python code and R specific. Preferably we would not > need a kludge like this :o > > > Maybe go with https://github.com/fgeorgatos/easybuild.experimental for > now to share it with others? > > That looks like a good place to start, so we can have a first version of an > pure > R easyconfig generator by the end of this week... > > > We should look into a contrib directory in the main EasyBuild repo > (https://github.com/hpcugent/easybuild), where people can get stuff > included without too much reviewing or very strict requirements, we already > have a couple of PRs for stuff like that (see > https://github.com/hpcugent/easybuild-framework/pull/1190, > https://github.com/hpcugent/easybuild-framework/pull/1180). > > Check, let me know when there is a contrib dir in place. > > Cheers, > > Pi > > > regards, > > > > Kenneth > > ------------------------------------------------------------- > phone: +31 6 143 66 783 > e-mail: [email protected] > skype: pieter.online > -------------------------------------------------------------
