On Mon, 16 Mar 2015 at 05:27 -0000, Fotis Georgatos wrote:
> first, sorry for reacting on this email so many days later, I wanted
> to give more time to it. In fact, it’s a lengthy response (it will
> be TL;DR for most others, be aware).
Thanks for the response and the useful information. Not too many
comments follow...(people should read the parent, I've deleted some
useful information in this response.)
> > My initial hope was that easybuild supported building software
> > outside of the live environment (mock, chroot, fakeroot). This is
> > not the case and I can live with it for now but I hope this
> > is/becomes part of the future direction.
>
> What you are saying, in effect, is to confine not only build
> process, but also build environment. I think all people who have
> been for a while in this business, have expressed consent to go in
> this direction (in fact, docker has been referenced quite a few
> times). fi. I’ve seen on IRC rjeschmi testing PRs from docker?!
This might be a good use case for docker. Based upon my minimal
understanding of docker most other use cases seem bizarre.
> To be fair, this may be beyond and/or orthogonal to EasyBuild:
>
> - noone blocks anybody to setup an isolated build environment under
> chroot or whatever However, this works well for one build, it is
> more tricky for builds with dependencies! IMHO it’s all doable, it
> just takes someone with a strong itch to jump on the problem.
It was/is a hope. If I ever get any free time at all (unlikely) I
might look at it further. As is, easybuild is useful for getting a
more structured application environment than we currently have (people
running things out of other peoples home directories).
> ie. the original EB authors try to disclaim responsibility for
> fitness of the builds “for any particular purpose”. I find that a
> fair game.
Agree, and with below.
> IMHO, as time advances and community knowledge is pooled together
> they would start looking less and less of examples and more and more
> of production configuration - for most. In fact, here is the current
> status summary:
>
> - If you have RHEL6, x86_64 w. AVX, Infiniband & CUDA - You have
> *near-production* builds
> - If you have alternative OS/architecture - you largely call them examples
No AVX or CUDA on our older larger cluster but that isn't currently a
problem. Otherwise, the current easyconfigs are helping a lot.
> One important trick, to help grow the community:
> a) invite your users to write easyconfigs/easyblocks
> b) if they can’t write easyconfigs/easyblocks, ask them to shell-script the
> build process
> c) it’s easy/deterministic to convert shell scripts to an EB build process
Most of this is unlikely with our current user base, but I will
attempt to allow for the possibility.
> Fact of life is, scientific software has a far too large
> configuration vector to handle and it is users’ responsibility to
> define that - for the reproducibility argument. More often than not,
> this includes the exact versions of software that should be
> employed.
I have tried to explain to people that for serious/important work they
should be building their own software stack to their specifications
and not depending upon system installed versions. Some software
changes almost daily (lammps, I'm looking at you) and the exact
version can matter.
However, easybuild does offer a decent environment with a degree of
consistency that may be adequate for most users. And for someone with
more exacting needs a private easybuild setup should be doable.
> OK. Over this spring, I expect to do a major rework over
> LifeScience/Bioinfo collection; if you are in this area hang on
> tight for major improvements and a worthy update.
I've produced my first "beta" version of an easybuild installation and
am interested in updating some of the software versions. Largely Life
Sciences, but significant other areas (fluid/molecular dynamics, a lot
of R, some C/C++, etc).
Just an example: my early attempts to upgrade even libpng ran into
issues where libharu seemed to really require the older version.
libharu already has a largish patch in easybuild which just needs to
be larger (or the package killed from whatever requires it which
looked possible at early glance).
> > Other things I plan to do are building documentation, demonstration,
> > example and performance cases for each major software package. In
> > part, this becomes site specific but I also hope for some support from
> > the HPCBIOS and similar efforts.
>
> I’d prefer that people see HPCBIOS more as a collection of RFC-like
> definitions (as IETF used to do):
> - common and well-defined configuration aspects of HPC sites, which
> do not constraint implementation
> In fact, two independent implementations which prove a compatibility
> layer make for a good HPCBIOS case! By no means do we suppose that
> all these definitions are good for all sites at once (ie. do
> mix’n’match).
They are something good to reference and can help set some of our
patterns.
> the standard IETF doctrine for RFCs (which I personally find very
> productive in the 90s) has been:
> - rough consensus and running code
>
> I hope we can achieve something similar in our field of interest.
I'm a co-author on one of the useless RFCs from 2001.
Stuart
--
I've never been lost; I was once bewildered for three days, but never lost!
-- Daniel Boone
