Sorry. I missed this useful response from last week.
On Tue, 10 Mar 2015 at 16:38 -0000, Kenneth Hoste wrote:
> On 10/03/15 18:18, Stuart Barkley wrote:
>
> > Is there a good way to see a dependency tree with easybuild (I'm
> > now running 2.0.0)? I'm trying to reconcile different dependency
> > requirements for my large build and am finding it difficult to
> > determine what packages are requiring other packages.
>
> Have you considered trying eb --dep-graph?
>
> That can be used to generate a .dot file describing the dependency
> graph (eb will tell you what's missing).
>
> With the right optional Python packages installed, you can also make
> it dump a rendered image (.png, .pdf).
>
> For example:
>
> eb --dep-graph=goolf.dot goolf-1.4.10.eb -r
I missed seeing this and I don't have the necessary dependencies
installed.
This would tend to imply that there is some dependency resolution
which could be useful. Is is possible to get the basic relationship
data without out sending things all the way through graphviz?
The graphviz dependencies are not installed in my easybuild version of
python which might be useful. I'll follow up on this in the future.
I'll leave the rant about useless modules being necessary just to
write a simple text mode .dot file aside for now (mostly).
> > Up until now, I've just been doing builds and when things die, fix
> > that problem and restart the build. This works after a fashion,
> > but takes a long time when there are multiple issues.
>
> What kind of problem are you running into, specifically?
For example: upgrading libpng across all my packages. I need to find
which packages directly or indirectly use it and rebuild them all (in
the correct order). If one package build fails, I would like the
build process to continue so I can find out if other packages are also
going to fail.
> > Also, once I've completed a build I need to go back are rebuild
> > everything again in case one of the later changes impacted an
> > earlier build.
>
> How could changes made later impact an earlier build?
After building several packages with the new libpng, I then find one
package which is just not able to handle the upgrade without more work
than I'm willing to do at the time (libharu) and want to back out and
go back with the first (or another different) version and need to
again rebuild things dependent upon the revised package.
[I know that easybuild can handle multiple library versions where
necessary, but I would prefer to keep things consistent.]
> > After a build is complete, I can do some grepping of the .eb files
> > in ebfiles_repo, but even this gets awkward.
>
> Grepping to do what exactly? Figure out what depends on what? If so,
> use --dep-graph, it's what you want.
The greping can get pretty ugly. I have something like the following
in my notes:
eb --configfiles app-c6-alpha.cfg --force --robot --dry-run --
easyconfigs/FDASCL_* easyconfigs/HPCBIOS_* easyconfigs/R-3.0.2-goolf-1.4.10.eb
> check-alpha.log
fgrep ' * [' check-alpha.log | cut -c8- | cut -d' ' -f1 | sort | xargs grep
libpng
I don't have the examples anymore, but early on I was using the above
eb command to get the full list of easyconfig files (including robot
dependencies). Then I would do the same --robot --dry-run build with
the full list and count the number of times an easyconfig file was
referenced. This would give me a reasonable dependency build order
(most referenced package first). I could then do the builds
individually in a shell loop in the right order without the --robot
option.
Some form of --dep-graph output is probably useful (a file compatible
with tsort would be best and probably almost identical to the .dot
file). I would prefer a flat text file instead of a .png file like I
usually see from graphviz stuff.
> > Part of the cause of this issue is that I'm creating a couple of
> > .eb files with the set of software I want to install (similar to
> > the HPCBIOS files). Currently, it appears that everything in a
> > single .eb file needs to use the same library versions and some of
> > the packages use different library versions (zlib, libpng, boost,
> > cmake).
>
> Yes, since all dependency specifications need to include strict
> versions specification, and by installing things via a single
> easyconfig file, you're basically saying all of those modules need
> to be loaded at the same time, so you can only have one version of
> each software package in there.
I saw 'easyblock = "Bundle"' in 2.0 and was hoping that was a new way
to just specify a bunch of packages to build without the full
dependency requirements applying at the bundle level. It seems to be
something else.
> Alternatively, you can just supply a directory of easyconfig files,
> rather than grouping them together in a single easyconfig file?
That might be possible, but my current easyconfig directory has a
number of other files which are still experimental. Also, some of the
easyconfig files I'm using are the distributed ones and I'm still
working out whether I want private copies of these files (especially
if they are identical to the distributed ones).
For a while I was also sourcing a configuration with different groups
of easyconfig files into my batch script (I do the actual builds on
our compute nodes).
> > Another useful option would be a --keep-going option to have
> > easybuild continue on with other packages even if one fails. I
> > would like to start a build before going home and have easybuild
> > do as much as possible.
>
> You can trigger this kind of behavior by using --dump-test-report,
> since EasyBuild will always try to put together a full test report
> for whatever if was specified to do.
I'll look at this.
> There should be a dedicated runtime option for this though, that's
> true. Should be fairly trivial (see the definition of
> 'exit_on_failure' in easybuild/main.py).
>
> This is only useful in case of building and installing multiple
> independent groups of software packages, of course, since as soon as
> building one dependency fails everything that required it
> will/should fail.
My test build has 120+ things in it (kitchen sink build). If gcc wont
build, things won't be good, but a lot of the dependency tree should
be pretty flat without too many interactions.
Part of what I'm looking for is the two extremes. There are things
that build fine that I will include without much thought. Then there
are things that are going to be a lot of trouble that I will triage as
to whether I want to try to fix or just throw out for initial support
(e.g. EMBOSS which depends upon libharu (apparently on life support)
which requires the old libpng).
Stuart
--
I've never been lost; I was once bewildered for three days, but never lost!
-- Daniel Boone
