Hi Stuart,
Thanks for sharing your experiences, this is always useful!
Some feedback inline.
On 08/06/16 19:15, Stuart Barkley wrote:
I had been away from easybuild for a while. Unfortunately I have
limited access access to the system where I'm bootstrapping easybuild
so may details are a little fuzzy.
I recently had problems similar to the parent message (but details
appear significantly different and resolved in another thread).
Vaguely: When I tried to re-bootstrap 2.7.0 I received errors in the
sanity check when trying to build a final 2.7.0. I needed to revert
to 2.4 or 2.5 in order to not get the sanity check error. Once
installing one of these older versions I was then able to forward
build up to 2.7 (the latest at that time).
Please check if you're still seeing these issues at all with an
up-to-date bootstrap scripts & the latest EasyBuild (v2.8.1), if you
have some time.
We do what we can to make the bootstrap procedure more robust, it should
be in a better state now compared to a couple of months ago...
After a little more experimentation, I found bootstrap life much
easier with the python-setuptools rpm installed. With that installed
I could build pretty much every version from every other version of
easybuild (2.2-2.7).
After getting easybuild fully bootstrapped, I don't think I needed
python-setuptools for future builds.
Setup:
- KVM virtual machine with internet access
- Stripped down CentOS 6.5 with limited additional packages
- tcl modules environment
I went through the same procedure as well (+ Lmod, optionally), and
documented/automated it thoroughly at https://github.com/boegel/eb_lmod .
It may be a bit outdated now though, in particular you currently need
setuptools as well.
- I use this VM to bootstrap easybuild into ~/.boot.
- I then use that easybuild to rebuild easybuild into my real
easybuild location.
Why not point the bootstrap script to that location instead?
The bootstrap already install EasyBuild with EasyBuild in the final
stage, you're just adding another EB-with-EB stage on top of that
basically...
- I also use the --stop=fetch option to use this easybuild instance
to download package sources for transfer to my internal build
system.
Observations:
- There may be some sort of silent partial dependency on
python-setuptools in the bootstrapping process (since about 2.5.0)
It may have helped with older versions of EasyBuild, but since v2.7.0
setuptools is a *hard* requirement, you'll run into error message if you
don't have setuptools available when trying to use EasyBuild v2.7 or newer.
This is clearly mentioned in the release notes
(http://easybuild.readthedocs.io/en/latest/Release_notes.html#v2-7-0-march-20th-2016),
and was also highlighted in the release announcement
(https://lists.ugent.be/wws/arc/easybuild/2016-03/msg00044.html).
I wasn't terribly happy with this at first, since it's kind of a
breaking change (and I'm not a super-big fan of setuptools...), but it
was the only way to properly deal with the 'easybuild' namespace problem
that was fixed in v2.7.0.
- I find it annoying that easybuild does not include a .eb file for
itself. It does have .eb files for previous versions, but when I want
to rebuild the current version I need to either "guess" what the .eb
file might look like or find it somewhere on github.
The main issue here is that I would need to include an easyconfig file
in the release that I can't (easily/reliably) test (chicken-or-egg
situation)...
Although I could at least try with a development version (or maybe a
release candidate).
If I automate this properly, I should be able to come up with a reliable
way of checking whether the easyconfig file will work once the release
is actually in place...
I've opened an issue to remind myself:
https://github.com/hpcugent/easybuild/issues/227 .
Once you have an EasyBuild that was installed with EasyBuild, you *have*
an easyconfig file for it in the EasyBuild installation directory itself
(see the 'easybuild*' subdirectory)...
The announcement messages include a pointer to the git version, but it
always takes me a while to find the right child link to get the raw
file.
The announcement for 2.8.1 gives the .eb file as:
<https://github.com/hpcugent/easybuild-easyconfigs/pull/3153/files>.
This is useful, but I need to chase a few more links to get to the raw
file (click on view, click on raw). Then I can copy/paste the URL
into a curl command on the build system. The raw file for this is at:
<https://raw.githubusercontent.com/boegel/easybuild-easyconfigs/296ccb4a05b7a7f1af64e72a7454696ea0154d52/easybuild/easyconfigs/e/EasyBuild/EasyBuild-2.8.1.eb>.
I think this is a static URL and would like to see it directly
included in the new version announcements.
OK, this is useful feedback indeed.
Usually you can find the easyconfig file for the latest EasyBuild in
https://github.com/hpcugent/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/e/EasyBuild
(but you'll still need to jump through some hoops to find the URL of the
raw version you can just curl); this is also mentioned in
http://easybuild.readthedocs.io/en/latest/Installation.html#updating-an-existing-easybuild-installation
which I usually just point to in a release announcement.
However, there's an easier way: using "--from-pr".
For example, to install EasyBuild v2.8.1 you can use "eb --from-pr 3153"
using EasyBuild v2.8.0.
I should probably mention this more clearly in the release announcement...
I used --from-pr so often that I just don't think about it anymore. :-)
Note that this doesn't require any setup at all w.r.t. GitHub
integration, since --from-pr is read-only (it basically is an integrated
curl in that respect); it should just work.
I'll try and make that more clear in future release announcements.
Thanks for the ongoing work with easybuild.
You're welcome, and please don't hesitate to provide feedback like this,
we can only remedy things if we're aware of problems/annoyances.
regards,
Kenneth
