Hi Ken, all, On Aug 6, 2013, at 2:23 PM, Kenneth Hoste wrote: > Some remarks below. Given the size of this message, maybe it's a good idea to > follow up the individual topics in separate threads with modified subjects?
Yeah, let's take off the ones that deserve a life of their own... ...I leave that to others' judgement though; here we go: >> Overall it works well and, you can entertain yourself by co-loading 70+ >> modules in one go: >> https://github.com/fgeorgatos/easybuild.experimental/blob/master/users/fgeorgatos/HPCBIOS/HPCBIOS_Bioinfo-goolf-1.4.10.eb >> Some packages still need a tweak to be mergeable in (ABySS, BLAT, BLAST, >> NCBI*, Trinity, TopHat) >> IMHO, using it just for building the software in one-go is probably of more >> practical use. > > What's missing here exactly? Simply easyconfigs that rely on biodeps? I'd say, nothing is missing, it's all working nicely. (strange, hm?) Remaining work is just about aligning the mentioned packages to biodeps, if possible; fi. Jens is about to work on TopHat, if he could make it biodeps-compatible, then _that_ would be nice (the dep to -extended has to change, as described in PR mentioned below) >> http://hpcbios.readthedocs.org/en/latest/HPCBIOS_2012-94.html [...] > Can you open an issue for an updated/extended biodeps easyconfig, and use > that to drive the discussion? OK, done: 1 PR for avoiding biodeps-extended as dependency (as builddependency is fine though) and 1 PR for refreshing/extending the list of HPCBIOS_2012-94 (#384 & #383 respectively) > How is the mirror-thing you were working on coming along? That seems to be a > criticial element in all this... I have got a good collection of tarballs, but I come to realize that you guys have prototyped on tarballs that exist only on your side (I think RAxML is one example) or, other funny situations where upstream changes the tarball without changing version or name (!) Rgputools was the later case. So, unless we define some procedure on handling this, there will be surprises. > Besides that: there's little we can do against software devs updating > released tarballs without bumping the version. > It's obviously wrong, but maybe then don't see it as a big problem. > Without checking for source 'correctness' in terms of MD5 sums or something > like that, there's no way in telling whether the sources you download are the > same ones that were used by the easyconfig author... Fully agreed. >> 4) arch-aware/distro-aware organization [...] > What do you want to organize here exactly? I'm missing context here... If we can come up with common codebase for the various arch/distro strings. >> 5) better organize build environment variables > [...] > The target to fix here is > https://github.com/hpcugent/easybuild-framework/issues/604 . > [...] > I agree this is an important issue, but since we (nor our users) are running > into problems related to this, we don't spend time on it for now. I am a little bit surprised about that: - The users at Ghent don't find themselves in surprise when fi. MPICC is defined to what corresponds to a GCC build instead of icc etc? >> 6) Implement common HPC policies [...] > If someone comes up with (versioned) HPCBIOS easyconfigs that group together > the must/should elements of particular HPCBIOS policies, we'll be happy to > include them in the easyconfigs repo and in the next EasyBuild release. > I don't see why not. Nice; fyi. I consider them in prototype most yet but, in practice I have started deploying them in production this month (ie. good enough to get some automation work done nicely). > My proposal: version these easyconfigs by date (e.g. > HPCBIOS_Bioinfo_2013-08.eb), and maybe provide separate ones for MUST and > MUST+SHOULD... OK, I get it: - versioning should be visible in the name, good point. - MUST vs MUST+SHOULD, alike biodeps vs biodeps-extended, alike R-bare vs R etc. > Having some kind of standard w.r.t. naming of easyconfigs and organization > would really help imho. Yes, I am already going towards that direction, see naming scheme here: https://github.com/fgeorgatos/easybuild.experimental/tree/master/users/fgeorgatos/HPCBIOS What would be your desires? can you propose other suggestions, to make them suitable for your needs? eg. Pick to comment for a start Math, Debuggers, LifeSciences. >> 7) Common list of (bioinformatics?) applications of interest [...] > The ideal situation would be that: > > a) someone who starts working on some app opens an issue first expressing > interest, and stating that he has started working on it > b) someome who wants to start working on some app checks whether an issue is > open first > > However, this requires discipline, and there's no way to enforce people to do > this (and hence, we don't do it ourselves). Yeah, I understand in theory that's how it should be done but I find first myself not doing it :) > Maybe we'll just have to live with it, and use it to our advantange, i.e. if > it turns out that two people are working on the same stuff, review each > others work, find a common ground and then either work together on it or let > one of both finish the job. > > I'm afraid there's no silver bullet here... > > What might help is an eb feature that allows one to crawl all EasyBuild > (experimental) repositories and branches in search for easyconfig files for a > particular software package/version. > But again, that requires discipline on the user: search first, start working > then (after opening an issue so others are aware). One of the early things i had toyed with (that funky `module search`, remember?) was to automatically do a `find` over a range of (mostly git) repos with easyconfigs; it worked quite well and helped me to organize bits on my end. What I was hoping for, would be as simple as a wiki table (that functions as a real wiki!) on which we can all start adding our wishlist, adding eg. initials at the end. That is super-useful when you have a half-baked easyconfig/easyblock and you want to invite people to help or test, because then you know exactly who cares about it. The other alternative is what I have been doing with opening issues over lists of applications, example https://github.com/hpcugent/easybuild-easyblocks/issues/82 , but that is a little bit embarrassing, because it is as if we go begging... :) Anyway, as long as you remain cool about it we can keep going like this for a while. thanks for arriving up to here :) , Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # Yelling in a CERN forum
