Jack, Fotis: Deferring to bottom posting as well, so get ready to scroll...
On 4/1/15, 6:01 PM, "Jack Perdue" <[email protected]> wrote: >Howdy Fotis (deferring to Fotis's preference to bottom posting), > >On 04/01/2015 05:54 PM, Fotis Georgatos wrote: >> Hi Jack, >> >> On Mar 31, 2015, at 5:01 PM, Jack Perdue <[email protected]> wrote: >>> Around these parts, our approach to R is: >>> >>> a) install one of the "default" versions in EB >>> (e.g. R-3.1.1-ictce-6.3.5-default-mt.eb) >>> which only installs the extensions that come >>> with R (as opposed to "bare" which excludes >>> them) >>> >>> b) in a non-EasyBuild tree, we install extensions >>> the usual R way, which takes care of the latest >>> versions and dependencies >>> >>> c) provide a module that loads the EB R-default >>> and sets a path to the extensions >>> >>> Seems to work nicely... >> OK, interesting. This may actually be a reproducible use-case. >> If it get¹s automated I¹d call that: R-ugly or R-...-1st_of_April :) >> (no offense to the TAMU work, you guys are doing your part just >>decently). >> Why not poke fun with some build processes which are just plainly >>insane?! >> >> F. >> > >Just a note from our in-house Nederlander, eR, R extension maintainer: > >----- >happy to hear we are doing our part "just decently" > >I'm not familiar how easybuild does it with the exts_list. So whenever >you want to install a new packages you have to add it to the exts_list >and then rebuild R (although it will skip the already installed parts)? >How does it handle packages that require non standard install? > >R has a very powerful easy-to-use build-in technique of installing new >packages (and yes it can be automated). To me it seems better to use a >specialized automated approach than to use a generic technique > >I think TACC is doing it in a similar way with installing packages >----- > >It provides flexibility... when a user needs an extension, done. > >The magic is just: > >prepend-path R_LIBS_USER >"/software/tamusc/R-packages/3.1.0-ictce-7.1.2-default-mt" > >(all EB software is in /software/easybuild). > >But just for you, Fotis, I'll see if I can come up with R-ugly using >the CmdCp easyblock. > >jack (who has some ideas in mind) This isn't about EB, but it might be helpful if Jack is planning to implement something similar. In spack, we opted for a happy medium for python extensions, and I suspect you could apply it to R. I don't think it's too terribly ugly. Spack installs all python modules in their own prefix, like EB does when not using exts_list. This allows you to have many different versions of the same package installed (e.g., several different numpy stacks). However, it is often nice to have some packages available automatically in a python installation, WITHOUT having to load a bunch of modules. You can really only have one version configured that way because otherwise they'll conflict (That is not entirely true since Python has some support for adding multiple versions of modules to the same python install, but client code has to know about it and request a version explicitly, and no one actually does that AFAIK). So, spack has support for activating and deactivating extensions: http://scalability-llnl.github.io/spack/basic_usage.html#extensions-python -support Activation *mostly* just symlinks the external module (and any dependency extensions) into the python install, but the python package itself can extend the way activation is done. That is used to provide some custom logic for merging easy_install.pth files and other things that would otherwise conflict within the same prefix. So, Spack python installs are a bit like virtualenvs, in that you can swap versions of extensions in and out by activating/deactivating them. So, for example, you can have as many numpys as you like installed, but only one can be "activated" in a particular python install at a time. Admins can activate the default, and import will just work. If an end user wants to use a different one, they can load a module for that, and it'll go in their PYTHONPATH, overriding the activated version. The extension mechanism is fairly generic so you could imagine implementing something similar for R: http://scalability-llnl.github.io/spack/packaging_guide.html#extensions With this kind of design, I *think* it would just requires overriding activate/deactivate in the R package, unless R has something MUCH more complicated than Python. -Todd
