Hi there, Guys, I think we have some good feedback in this email.
Contributions, Questions, Wishlist; it just has it all ;-) 1) Contribution: Bioinformatics and extra packages Together with the help of Cedric Laczny (it means he did most of the job ;-), we can announce an initial round of .eb packages of bioinformatics tools: https://github.com/fgeorgatos/easybuild/tree/master/experimental MUMmer-3.23-goalf-1.1.0-no-OFED.eb MetaVelvet-1.2.01-goalf-1.1.0-no-OFED.eb RNAz-2.1-goalf-1.1.0-no-OFED.eb SOAPdenovo-1.05-goalf-1.1.0-no-OFED.eb ViennaRNA-2.0.7-goalf-1.1.0-no-OFED.eb amos-3.1.0-goalf-1.1.0-no-OFED.eb boost-1.50.0-goalf-1.1.0-no-OFED-static.eb # this may require revision boost-1.50.0-goalf-1.1.0-no-OFED.eb bowtie2-2.0.0-beta7-goalf-1.1.0-no-OFED.eb bwa-0.6.2-goalf-1.1.0-no-OFED.eb clustalw2-2.1-goalf-1.1.0-no-OFED.eb fastx_toolkit-0.0.13.2-goalf-1.1.0-no-OFED.eb idba_ud-1.0.9-goalf-1.1.0-no-OFED.eb infernal-1.0.2-goalf-1.1.0-no-OFED.eb libgtextutils-0.6.1-goalf-1.1.0-no-OFED.eb oases-0.2.08-goalf-1.1.0-no-OFED.eb pandaseq-2.1-goalf-1.1.0-no-OFED.eb samtools-0.1.18-goalf-1.1.0-no-OFED.eb velvet-1.2.07-goalf-1.1.0-no-OFED.eb Our team will produce more and announce them as they become available. I have also somewhere else these experiments, which seem to function OK: GCC-4.7.1.eb CMake-2.8.8-goalf-1.1.0-no-OFED.eb octave-3.6.2-goalf-1.1.0-no-OFED.eb If interested in any, drop a note. 2) Contribution, part #2: package descriptions generated via pkg2eb OK, this is a wild ride. I had a little bit too much time yesterday afternoon and decided to peek & poke on NetBSD's pkgsrc. The Makefiles contain much of the information needed to feed easybuild with and, a tool was born... pkg2eb! It is a blind robot really: it will try to best-guess the possible parameters; I am still debugging it and a release should rather wait, but here's the first bunch of builds that were done fully automated: (only some of the modules are interesting; yes, there are no sanity checks...) ------------- /home/clusterusers/fgeorgatos/easybuild/modules/all/ ------------- SPTK-3.5/3.5-goalf-1.1.0-no-OFED-20120724 a52dec/0.7.4-goalf-1.1.0-no-OFED-20120724 advancecomp/1.15-goalf-1.1.0-no-OFED-20120724 alsa/lib-1.0.22-goalf-1.1.0-no-OFED-20120724 bladeenc/0.94.2-goalf-1.1.0-no-OFED-20120724 bml/0.5.0-goalf-1.1.0-no-OFED-20120724 cabextract/1.4-goalf-1.1.0-no-OFED-20120724 celt/0.5.0-goalf-1.1.0-no-OFED-20120724 dact/0.8.41-goalf-1.1.0-no-OFED-20120724 dar/2.4.1-goalf-1.1.0-no-OFED-20120724 fcrackzip/1.0-goalf-1.1.0-no-OFED-20120724 fluidsynth/1.1.5-goalf-1.1.0-no-OFED-20120724 goom/2004.0-goalf-1.1.0-no-OFED-20120724 lame/3.99.5-goalf-1.1.0-no-OFED-20120724 libcdaudio/0.99.12-goalf-1.1.0-no-OFED-20120724 libcddb/1.3.2-goalf-1.1.0-no-OFED-20120724 libdca/0.0.5-goalf-1.1.0-no-OFED-20120724 libdiscid/0.2.2-goalf-1.1.0-no-OFED-20120724 libid3tag/0.15.1b-goalf-1.1.0-no-OFED-20120724 libifp/1.0.0.2-goalf-1.1.0-no-OFED-20120724 liblzo-1.08/1.08-goalf-1.1.0-no-OFED-20120724 libmodplug/0.8.8.4-goalf-1.1.0-no-OFED-20120724 libmp3splt/0.7-goalf-1.1.0-no-OFED-20120724 libmpd/0.20.0-goalf-1.1.0-no-OFED-20120724 libmpdclient/2.6-goalf-1.1.0-no-OFED-20120724 libsamplerate/0.1.7-goalf-1.1.0-no-OFED-20120724 libsidplay/1.36.59-goalf-1.1.0-no-OFED-20120724 libsmf/1.3-goalf-1.1.0-no-OFED-20120724 libsndfile/1.0.25-goalf-1.1.0-no-OFED-20120724 libvisual/0.4.0-goalf-1.1.0-no-OFED-20120724 libvisual0.2-0.2.0/0.2.0-goalf-1.1.0-no-OFED-20120724 libzip/0.10.1-goalf-1.1.0-no-OFED-20120724 lzip/1.13-goalf-1.1.0-no-OFED-20120724 lziprecover/1.13-goalf-1.1.0-no-OFED-20120724 lzmalib/0.0.1-goalf-1.1.0-no-OFED-20120724 mac/3.99u4b5s7-goalf-1.1.0-no-OFED-20120724 malint/0.2-goalf-1.1.0-no-OFED-20120724 mp3blaster/3.2.5-goalf-1.1.0-no-OFED-20120724 mp3check/0.8.3-goalf-1.1.0-no-OFED-20120724 mp3wrap/0.5-goalf-1.1.0-no-OFED-20120724 mserv/0.35-goalf-1.1.0-no-OFED-20120724 normalize/0.7.7-goalf-1.1.0-no-OFED-20120724 opencore/amr-0.1.2-goalf-1.1.0-no-OFED-20120724 portaudio/devel-20060521-goalf-1.1.0-no-OFED-20120724 rzip/2.1-goalf-1.1.0-no-OFED-20120724 shorten/3.6.1-goalf-1.1.0-no-OFED-20120724 snd/9.9-goalf-1.1.0-no-OFED-20120724 sox/14.3.2-goalf-1.1.0-no-OFED-20120724 speex/1.2rc1-goalf-1.1.0-no-OFED-20120724 szip/2.1-goalf-1.1.0-no-OFED-20120724 tar/1.26-goalf-1.1.0-no-OFED-20120724 unshield/0.5.1-goalf-1.1.0-no-OFED-20120724 xmms/faad-2.7-goalf-1.1.0-no-OFED-20120724 xz/5.0.3-goalf-1.1.0-no-OFED-20120724 zutils/0.9-goalf-1.1.0-no-OFED-20120724 3) Question on Python options On 11/07/2012 13:44, Stijn De Weirdt wrote: > there is no easy way here: eg the Python install (unless i'm mistaken) comes > with mpi4py, so you need a mpi stack, and numpy needs blas and fft(w). what is > currently in a toolkit is "minimal" build environment OK, to my pleasant surprise, the Python builds include numpy & scipy; Of course, that explains it all: you certainly need to specify the compiler. Related question: are there going to be separate builds for numpy & scipy? If yes, someone should be able to mix Python, numpy, scipy versions etc!? And since Python v3 requires a manual hack to build on Debian (dep: tcsh), this bring us to the next topic... 4) Wishlist So, while using the tool we've come with some desired features: https://github.com/fgeorgatos/easybuild/wiki/Wishlist """ Here is a list of features that are desired in next versions of EasyBuild: USER-oriented features: * Consider category-oriented module sets (eg. `module load bioinformatics-tools`) # may be done outside of EB logic * Allow for custom-definition of easyblock paths (instead of enforcing any "a-z/" dir structure * How to organize a collection of .eb files coming from different sources? (eg. distinct git repos) * Seen at http://joeyh.name/code/mr/ : "When updating a git repository, pull from two different upstreams and merge the two together." SYSADMIN-oriented features: * Provide some smart way to handle dependencies like "tcsh" which have no meaning on Debian * Define the namespace "by a standard"; eg. like [https://github.com/fgeorgatos/HPC-RFC/blob/master/0001/0001.md] * Give freedom to define the namespace format (eg. original Package names OR the lower case version) * Support more configurable formats of version strings (as seen below, how other HPC sites do it) * nested levels (ie. split version string in sub sections) * provide hooks for gnu/gcc/intel/pgi etc strings """ It may be that solutions can be provided outside of EasyBuild itself or an FAQ could give the obvious answers, if there are. One thing that is worthy to point out is the modules namespace & Easyblocks; The following, in effect prevents me from writing the fancy language "r": class_name = name[0].upper() + name[1:].replace('-','_') Yes, I mean "r" vs "R" ;-) I can accept the logic of needing to create a case-insensitive namespace for clarity on the user side but, is there another systems reason for it? If we can reply that last question, then the whole discussion can move on to the other SYSADMIN-oriented features, which in a way derive from that. My stance is, that EasyBuild has all the foundations to become a standard tool for many more HPC sites but, it must give freedom to the sysadmin. 5) Epilogue Finally, I am experimenting with some concepts on "freezing" github repos and creating running software for that; if someone else is doing something related, please keep in touch ;-) cheers, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # Yelling in a CERN forum
