Hi Stuart, On Feb 11, 2015, at 2:32 AM, Stuart Barkley <[email protected]> wrote: > I've been looking at easybuild for a little while now and it appears > to be a suitable framework for a good portion of our needs.
If you provide software for HPC you are on the right road. > I think I eventually used "--dry-run --robot" and grepped the output > to determine the full dependency lists and create individual 'eb > --stop=fetch --force' commands for every package dependency. I also > used something like this to determine an internal build order so I > didn't need to --robot 50 packages which would all churn and fail on > the same dependency. I think this is the best you can do to pre-download sources nowadays, if case you don’t provide continuous network access, as you described. > What do people do/recommend for multiple OS environments? We are > currently CentOS 6 but will eventually move to C7. I'm thinking I > will want a separate application tree for each OS (/projects/app-c6 > and /projects/app-c7). > > How do people deal with software with frequent updates (java) or > security issues? Do you rebuild and remove old packages? You may be able to handle both of the above needs, by using the concept of buildsets, mentioned in p. over here: https://archive.fosdem.org/2014/schedule/event/hpc_devroom_hpcbios/attachments/slides/491/export/events/attachments/hpc_devroom_hpcbios/slides/491/FOSDEM14_HPC_devroom_09_HPC_BIOS.pdf In principle, the idea is that you create self-contained directory areas with complete build trees, including modules, at a given point in time. I’ve calling them /opt/apps/HPCBIOS.YYYYMMDD but any kind of tag will just do. Then you might create symlinks like: /opt/apps/sandybridge -> /opt/apps/*.YYYYMMDD I used a dubious example name above but you get the idea. The point is that you modify your symlinks when you transit maintenance windows, so that you don’t break your running jobs. > Do people recommend updating toolchains and the dependencies (e.g. > eliminate multiple versions of python 2.7)? Since this will be our > initial build I don't really need python 2.7.3, 2.7.5 or .2.7.6. > Should I just use the --try options to build for 2.7.8? If you do life sciences/bioinformatics you’ll likely need Python/2.7.3; you can always try to upgrade forward towards whatever, however I’d advise you to do this as a secondary stage, or you’ll be distracted by compatibility issues that raise quickly in these situations. > information about the 1.7 toolchain, I need to review that email > thread. I’m the perpetrator; it’s basically a GCC/4.8.x fresh toolchain, which should serve for a little while. If I was you , I would do the above step and then try something like: eb HPCBIOS_LifeSciences*eb --try-toolchain=goolf,1.7.20 -r That should provide for a quite interesting effect :) Feel free to depart then with other improvements, you may do plenty. > Is there any planned support for beta/test packages? At the present > time I plan to have two separate easybuild installations and after > building/testing an application in the beta installation I'll rebuild > it in the production installation. (With C6 and C7 installations this > might multiply to 4 easybuild installations.) I am afraid that yes, it’s the only way, if you want to play safe. A rationalized way to handle it, is to create “future” buildsets which you test extensively before a maintenance window; once you arrive at the crossing point, you move symlinks around. You can easily roll-back and roll-forward on the cheap, in this way. > Is there any planned support (or something I'm missing) for allowing > someone else to use an existing easybuild installation (including > toolchains and other packages) to build a test/prototype package? I > would like to have other staff be able to build local test packages > off of the production toolchains and be able to give me a set of > proposed/working .eb files for final build. This is similar to the > above beta question, except I don't expect others to be able to > maintain their own easybuild installation (and don't want them writing > into the production installation). Try MODULEPATH=$MODULES_USERAREA:$MODULES_SYSTEMAREA You may also add $MODULES_GROUPAREA in the mix, you get the idea. Many HPC users can actually maintain their easyconfigs, quite well. They are having much bigger problems to solve, most of the time :) If they are python-adverse though, just invite them to write shell scripts and then it’s trivial to convert them to the EB world of things. > Thanks for the work on easybuild. This looks like it will address a > long standing need for building stable versions of software for our > users. The best thanks is sharing easyconfigs ;-) enjoy, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum
