On Tue, 2012-07-24 at 19:48 +0200, Fotis Georgatos wrote: > Hi there, > > Guys, I think we have some good feedback in this email. > > Contributions, Questions, Wishlist; it just has it all ;-) > > > 1) Contribution: Bioinformatics and extra packages > > Together with the help of Cedric Laczny (it means he did most of the job ;-), > we can announce an initial round of .eb packages of bioinformatics tools: > https://github.com/fgeorgatos/easybuild/tree/master/experimental
> > Our team will produce more and announce them as they become available. > I noticed this yesterday, looks great and we're very happy you're contributing back! > I have also somewhere else these experiments, which seem to function OK: > GCC-4.7.1.eb > CMake-2.8.8-goalf-1.1.0-no-OFED.eb > octave-3.6.2-goalf-1.1.0-no-OFED.eb > > If interested in any, drop a note. Please don't hesitate to add them to your github repo and create a pull request for it. (to our develop branch please) > > > 2) Contribution, part #2: package descriptions generated via pkg2eb > > OK, this is a wild ride. I had a little bit too much time yesterday afternoon > and decided to peek & poke on NetBSD's pkgsrc. The Makefiles contain much of > the information needed to feed easybuild with and, a tool was born... pkg2eb! > > It is a blind robot really: it will try to best-guess the possible parameters; > I am still debugging it and a release should rather wait, but here's the first > bunch of builds that were done fully automated: > > (only some of the modules are interesting; yes, there are no sanity checks...) These can indeed be a very good starting point to create your EasyBlocks and EasyConfigs from. We also noticed a lot of practical information on building software can be found in Homebrew (for mac) [0] and of course gentoo packages. But we noticed these are not focused on HPC software and almost always require manual changes/patches to get them working with the Intel compiler and the $SOFTROOTXXX instead of just looking in /usr/include ... They do not need manual changes when it's a simple configure make make install, but these are very easy for us to generate in any way. Still, might be useful if EasyBuild gains a broader user base. > > 3) Question on Python options > > Are there going to be separate builds for numpy & scipy? > If yes, someone should be able to mix Python, numpy, scipy versions etc!? > numpy and scipy are currently installed as an extention of ApplicationPackage, so they get installed inside the site-packages directory of the python version. Like we also do for gem's in ruby and packages in R. But we also provide a PythonPackageModule which you can extend to build numpy and scipy (and any other python package) as a separate module. So yes, you could build different versions of scipy and numpy to be used without having to rebuild Python this way. > 4) Wishlist > > So, while using the tool we've come with some desired features: > https://github.com/fgeorgatos/easybuild/wiki/Wishlist > """ > Here is a list of features that are desired in next versions of EasyBuild: > > USER-oriented features: > * Consider category-oriented module sets (eg. `module load > bioinformatics-tools`) # may be done outside of EB logic You can just create a .eb file with no sources, but only dependencies on other modules, This will then create a module that when loaded will also load all it's dependencies. (Or just create these modulefiles yourself) > * Allow for custom-definition of easyblock paths (instead of enforcing any > "a-z/" dir structure The a-z directory structure is not enforced. EasyBuild will just look into the python path if it can finde a python module with the correct name and also try to find it in a a-z directory. > * How to organize a collection of .eb files coming from different sources? > (eg. distinct git repos) We are currently already using disting git repos to get our easyblocks from (our public and private repo) Again, just make sure the easyblocks are in the python path and everything should be detected. This is why every directory has a __init__.py file with this content: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) This declares a namespace package, so python will continue to search in other place on the path if it can not find e.g., Python in the /p directory of the first repo. > * Seen at http://joeyh.name/code/mr/ : "When updating a git repository, pull > from two different upstreams and merge the two together." We do not need something like this, as said above, just make sure the python path is set correctly and EasyBuild will find the files. > > SYSADMIN-oriented features: > * Provide some smart way to handle dependencies like "tcsh" which have no > meaning on Debian This is indeed a problem, we added detection for debian packages with dpkg in the develop branch now. But since packages across distro's can have different names this is no perfect solution. A way to solve this would be by just adding a .eb file for tcsh and treat it as a normal dependency, but we don't actually want to build an entire linux distro this way. This also would not work for kernel modules which are a real system dependency (e.g., for infiniband support in certain mpi versions) So feedback here is welcome. > * Define the namespace "by a standard"; eg. like > [https://github.com/fgeorgatos/HPC-RFC/blob/master/0001/0001.md] > * Give freedom to define the namespace format (eg. original Package names OR > the lower case version) > * Support more configurable formats of version strings (as seen below, how > other HPC sites do it) > * nested levels (ie. split version string in sub sections) Agreed, we're going to add a configuration options for this. > > * provide hooks for gnu/gcc/intel/pgi etc strings What exactly do you mean here? > It may be that solutions can be provided outside of EasyBuild itself or an FAQ > could give the obvious answers, if there are. > I will make sure my explanations here get added to our wiki and/or a FAQ section. And we're also in the process of completing them before the 1.0 release. > One thing that is worthy to point out is the modules namespace & Easyblocks; > The following, in effect prevents me from writing the fancy language "r": > class_name = name[0].upper() + name[1:].replace('-','_') > Yes, I mean "r" vs "R" ;-) > I can accept the logic of needing to create a case-insensitive namespace > for clarity on the user side but, is there another systems reason for it? You could always create a class 'R_' and use easyblock="R_" in your .eb file. You are correct that this will not work by default however. The reason here is that we try to adhere to PEP8 where possible, which declares [1]: Almost without exception, class names use the CapWords [2] convention. If you believe this can be considered an exception we can leave it out again. > > If we can reply that last question, then the whole discussion can move on > to the other SYSADMIN-oriented features, which in a way derive from that. > > My stance is, that EasyBuild has all the foundations to become a standard > tool for many more HPC sites but, it must give freedom to the sysadmin. We greatly appreciate your feedback and are very happy this is proving a useful tool for you. > 5) Epilogue > > Finally, I am experimenting with some concepts on "freezing" github repos > and creating running software for that; if someone else is doing something > related, please keep in touch ;-) What exactly do you mean by freezing? > > cheers, > Fotis > Jens Timmerman [0] https://github.com/mxcl/homebrew/tree/master/Library/Formula [1] http://www.python.org/dev/peps/pep-0008/#class-names [2] http://en.wikipedia.org/wiki/CamelCase
