Hi all, IMHO the best solution to this issue is to sync the "sources" dir between all sites. This is feasible, if you consider that at CyI we currently have 55G of sources and there are a lot of packages there :)
Best regards, *Andreas Panteli* HPC Systems Support. CaSToRC. The Cyprus Institute *Tel:* +357 22208610 *Email:* [email protected] On 4 December 2013 11:14, Kenneth Hoste <[email protected]> wrote: > Hi all, > > First of all: welcome to the wonderful world of EasyBuild Raik! And thanks > for not giving up on it instantly after something went wrong... > > > Sometimes this problem occurs not because of a parallel build (like Fotis > mentioned), but because you cancelled the 'eb' run because you realized you > wanted to change something (e.g. enable debug logging, or whatever). > > If you cancel eb while it's downloading stuff, you'll corrupt the file > that's being downloaded at that point. > In that case: simply remove that file, start EB again. > > Stijn: it's not always possible to make EB clean up 'partial downloads', > e.g. when it's cancelled or when the connection goes down (well, maybe then > it is possible, depends). > > We can resort to MD5 sums though, but that's a huge effort, and I'm pretty > sure there be dragons there too (e.g. upstream sources that get changed, > like rgputools, WPS, ...). > > > regards, > > Kenneth > > On 04 Dec 2013, at 10:05, Fotis Georgatos wrote: > > > > > Hi Raik, > > > > On Dec 4, 2013, at 9:42 AM, <[email protected]> < > [email protected]> wrote: > >> parse_cmd_output): cmd "tar xjf /home/eb/.local/easybuild/sources/g/GCC/ > >> gmp-5.0.4.tar.bz2" exited with exitcode 2 and output: > >> bzip2: (stdin) is not a bzip2 file. > >> tar: Child returned status 2 > >> tar: Error is not recoverable: exiting now > >> --- > >> > >> Somebody an idea? > > > > I often get these kind of errors as well, it universally reduces to the > following situation: > > * the cause is my bad, because if running multiple concurrent builds > (via xargs or parallel) > > * I manually try to bunzip/untar, to verify that the sourceball is > indeed corrupted > > * generally it is, so I simply remove it, and run eb exactly once, to > re-download it > > * eb should go through the extract step unobstructed, that's your green > light > > * by this moment you can now spawn 10+ builds over the same source w/out > issues > > > > The good news are, that as you progress in the effort over multiple > weeks, > > and build your "source" dir, it becomes less and less probable to hit > this issue. > > > > Potential solutions for the community: > > * (short-term) add md5/sha hashes, to warn the user gracefully about > this issue > > * (long-term) use some scheduling of short that prevents parallel > downloads > > (ie. either make each write step atomic or, simply re-schedule builds) > > > > I am totally interested to heat if anybody has to propose a way > > to avoid certain parallel builds, because it also solves some other > issues > > (some packages eg. NCBI vs BLAST do not like to be build alongside each > other) > > > > cheers, > > Fotis > > > > -- > > echo "sysadmin know better bash than english" | sed s/min/mins/ \ > > | sed 's/better bash/bash better/' # Yelling in a CERN forum > > > > > > > > > > >
