hi fotis,
first of all thanks for the feedback, interest and usage. glad to see
you can pick it up in such short time.
wrt the wishlist, we have an internal trac ticketing system for our
current features and bugs, maybe we should expose that to outside , so
you can add your request there or at least have an idea what is
coming/being worked on (i'll leave this to jens and kenneth to decide).
a small remark wrt the eb and pkg2eb: i don't think it is a race of
adding as much possible packages to a list. easybuild will not gain
support if we can "only" do default gcc based compilation. we have
software repositories for that ;).
a first step in providing easybuild support should be that one reads
(and tries to understand) the configure options and dependencies of a
package. second, one should try to build with something else then gcc
(eg intel): it will show how good the default package is wrt
building/installing and will identify which packages need an easyblock.
> Our team will produce more and announce them as they become available.
we will see if we can send you a copy of our not-yet publsihed .eb and
old style blocks, to avoid duplication of work and to speedup the
porting to the new version.
2) Contribution, part #2: package descriptions generated via pkg2eb
OK, this is a wild ride. I had a little bit too much time yesterday afternoon
and decided to peek & poke on NetBSD's pkgsrc. The Makefiles contain much of
the information needed to feed easybuild with and, a tool was born... pkg2eb!
It is a blind robot really: it will try to best-guess the possible parameters;
I am still debugging it and a release should rather wait, but here's the first
bunch of builds that were done fully automated:
(only some of the modules are interesting; yes, there are no sanity checks...)
such an automated build is a very good first step, but indeed more
checking should be added before we should give it the "supported by
Easybuild" label (imho).
what also could be useful (i think there's an internal ticket for that)
is that based on such source packages (eg like the .src.rpm for rpm
based systems), you have an easyblock that takes these source packages
as source and builds and install them as they were intended by the
distribution. and since you are a debian expert, we also need an
easyblock to install binary .deb in easybuild (we have that for rpms,
it's how java is installed atm).
3) Question on Python options
On 11/07/2012 13:44, Stijn De Weirdt wrote:
there is no easy way here: eg the Python install (unless i'm mistaken) comes
with mpi4py, so you need a mpi stack, and numpy needs blas and fft(w). what is
currently in a toolkit is "minimal" build environment
OK, to my pleasant surprise, the Python builds include numpy & scipy;
Of course, that explains it all: you certainly need to specify the compiler.
Related question: are there going to be separate builds for numpy & scipy?
If yes, someone should be able to mix Python, numpy, scipy versions etc!?
this is ofcourse a problem, we already have that with toolkits. module
load Python in easybuild actually means module load
Python/someversion-numpy-someversion-scipy-someversion-matplotlib-someversion-mpi4py-someversion...
this is a matter of organising i think: i prefer a good, well featured
base version, with some possible exceptions so you would have to do
module load Python/vX-1
module load Python-scipy/vX-Python-vX-1 (i have no clue how this should
be organised or named)
over
module load Python/x numpy/y scipy/z matplotlib/v1 mpi4py/v2 ...
anyway, i don't think there's a silver bullet here.
And since Python v3 requires a manual hack to build on Debian (dep: tcsh),
this bring us to the next topic...
4) Wishlist
So, while using the tool we've come with some desired features:
https://github.com/fgeorgatos/easybuild/wiki/Wishlist
"""
Here is a list of features that are desired in next versions of EasyBuild:
USER-oriented features:
* Consider category-oriented module sets (eg. `module load
bioinformatics-tools`) # may be done outside of EB logic
what should this do: load modules or change the modulepath so a module
available becomes readable again
* Allow for custom-definition of easyblock paths (instead of enforcing any
"a-z/" dir structure
* How to organize a collection of .eb files coming from different sources?
(eg. distinct git repos)
* Seen at http://joeyh.name/code/mr/ : "When updating a git repository, pull
from two different upstreams and merge the two together."
SYSADMIN-oriented features:
* Provide some smart way to handle dependencies like "tcsh" which have no
meaning on Debian
* Define the namespace "by a standard"; eg. like
[https://github.com/fgeorgatos/HPC-RFC/blob/master/0001/0001.md]
* Give freedom to define the namespace format (eg. original Package names OR
the lower case version)
* Support more configurable formats of version strings (as seen below, how
other HPC sites do it)
* nested levels (ie. split version string in sub sections)
* provide hooks for gnu/gcc/intel/pgi etc strings
"""
i'm missing something here: you have some concrete examples?
i think what could be interesting is that the default easyblock is as
complete as possible, and that we have an easy way to generate symlinks
to these module files so anyone can use there own naming convention
It may be that solutions can be provided outside of EasyBuild itself or an FAQ
could give the obvious answers, if there are.
One thing that is worthy to point out is the modules namespace & Easyblocks;
The following, in effect prevents me from writing the fancy language "r":
class_name = name[0].upper() + name[1:].replace('-','_')
Yes, I mean "r" vs "R" ;-)
;)
I can accept the logic of needing to create a case-insensitive namespace
for clarity on the user side but, is there another systems reason for it?
we tried to follow some PEP standard
If we can reply that last question, then the whole discussion can move on
to the other SYSADMIN-oriented features, which in a way derive from that.
My stance is, that EasyBuild has all the foundations to become a standard
tool for many more HPC sites but, it must give freedom to the sysadmin.
i can understand the freedom on the actual enduser part (ie the module
naming or renaming), but what other degrees of freedom are you thinking of?
5) Epilogue
Finally, I am experimenting with some concepts on "freezing" github repos
and creating running software for that; if someone else is doing something
related, please keep in touch ;-)
anyway, thanks for trying it out and giving feedback. it's much appreciated!
stijn
cheers,
Fotis
