Dear Kenneth, I had easyconfigs that I used on an old cluster mixed into the new easyconfig files which I was installing on a new cluster, which is why the files were still from EasyBuild 3.x. I have removed all the old easyconfigs that I had included and am now remaking them using the new easyconfig info. For the easyconfig that I had the previous problem this fixes it.
Thanks a lot for your help, Niek On Tue, Nov 19, 2019 at 4:40 PM Kenneth Hoste <kenneth.ho...@ugent.be> wrote: > Dear Niek, > > On 19/11/2019 22:28, Niek de Klein wrote: > > Dear Kenneth, > > > > Attached the log. > > Thank you, that's helpful to figure this out. > > So, here's what's going on... > > When resolving dependencies for M4-1.4.17-foss-2018b.eb, EasyBuild goes > looking for the easyconfigs for the subtoolchains of foss/2018b, > starting with gompi/2018b. > > It finds an easyconfig for gompi/2018b at > > /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/g/gompi/gompi-2018b.eb > > This easyconfig file specifies that gompi/2018b consists of GCC 4.9.3, > binutils 2.25 and OpenMPI 1.8.8, which is very incorrect... > > This makes things go horribly wrong when it tries to find the easyconfig > for OpenBLAS, eventually. > > > So, question is why that gompi-2018b.eb easyconfig file isn't what it's > supposed to be (see > > https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/g/gompi/gompi-2018b.eb > ). > > The warnings you're hitting also show that you're using a copy of the > easyconfig files from an EasyBuild 3.x version. > > The easyconfigs included in EasyBuild 4.0.x have been cleaned up to > avoid those warnings... > > > > regards, > > Kenneth > > > > On Tue, Nov 19, 2019 at 4:00 PM Kenneth Hoste <kenneth.ho...@ugent.be > > <mailto:kenneth.ho...@ugent.be>> wrote: > > > > Dear Niek, > > > > On 19/11/2019 20:42, Niek de Klein wrote: > > > I get > > > > > > $ eb M4-1.4.17-foss-2018b.eb -D > > > > > > Please try running this with debugging enabled, and then zip up the > log > > file from /tmp (see path printed at the line like shown below): > > > > eb --debug M4-1.4.17-foss-2018b.eb -D > > > > > > > == temporary log file in case of crash > > /tmp/eb-ZCIA8v/easybuild-JS80fY.log > > > > > > > > <removed a bunch of warnings> > > > > > > ERROR: Failed to process easyconfig > > > > > > > /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/s/ScaLAPACK/ScaLAPACK-2.0.2-gompi-2018b-OpenBLAS-0.3.1.eb: > > > > > Failed to determine minimal toolchain for dep OpenBLAS > > 0.3.1-GCC-7.3.0-2.30 > > > > > > and a bunch of warnings (full log below). > > > > > > eb -S OpenBLAS-0.3.1-GCC-7.3.0-2.30 > > > > > > gives > > > > > > > > > > CFGS1=/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/o/OpenBLAS > > > * $CFGS1/OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb > > > > > > and > > > > > > eb OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb -D > > > == temporary log file in case of crash > > /tmp/eb-VAv5nH/easybuild-9UafJT.log > > > Dry run: printing build status of easyconfigs and dependencies > > > > > > > CFGS=/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs > > > * [x] $CFGS/m/M4/M4-1.4.18.eb (module: M4/1.4.18) > > > * [x] $CFGS/z/zlib/zlib-1.2.11.eb (module: zlib/1.2.11) > > > * [x] $CFGS/h/help2man/help2man-1.47.4.eb (module: > help2man/1.47.4) > > > * [x] $CFGS/m/M4/M4-1.4.17.eb (module: M4/1.4.17) > > > * [x] $CFGS/b/Bison/Bison-3.0.4.eb (module: Bison/3.0.4) > > > * [x] $CFGS/f/flex/flex-2.6.4.eb (module: flex/2.6.4) > > > * [x] $CFGS/b/binutils/binutils-2.30.eb (module: binutils/2.30) > > > * [x] $CFGS/g/GCCcore/GCCcore-7.3.0.eb (module: GCCcore/7.3.0) > > > * [x] $CFGS/z/zlib/zlib-1.2.11-GCCcore-7.3.0.eb (module: > > > zlib/1.2.11-GCCcore-7.3.0) > > > * [x] $CFGS/h/help2man/help2man-1.47.4-GCCcore-7.3.0.eb (module: > > > help2man/1.47.4-GCCcore-7.3.0) > > > * [x] $CFGS/m/M4/M4-1.4.18-GCCcore-7.3.0.eb (module: > > > M4/1.4.18-GCCcore-7.3.0) > > > * [x] $CFGS/b/Bison/Bison-3.0.4-GCCcore-7.3.0.eb (module: > > > Bison/3.0.4-GCCcore-7.3.0) > > > * [x] $CFGS/b/Bison/Bison-3.0.5-GCCcore-7.3.0.eb (module: > > > Bison/3.0.5-GCCcore-7.3.0) > > > * [x] $CFGS/f/flex/flex-2.6.4-GCCcore-7.3.0.eb (module: > > > flex/2.6.4-GCCcore-7.3.0) > > > * [x] $CFGS/b/binutils/binutils-2.30-GCCcore-7.3.0.eb (module: > > > binutils/2.30-GCCcore-7.3.0) > > > * [x] $CFGS/g/GCC/GCC-7.3.0-2.30.eb (module: GCC/7.3.0-2.30) > > > * [x] $CFGS/o/OpenBLAS/OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb (module: > > > OpenBLAS/0.3.1-GCC-7.3.0-2.30) > > > == Temporary log file(s) /tmp/eb-VAv5nH/easybuild-9UafJT.log* > > have been > > > removed. > > > == Temporary directory /tmp/eb-VAv5nH has been removed. > > > > > > == temporary log file in case of crash > > /tmp/eb-ZCIA8v/easybuild-JS80fY.log > > > > > > WARNING: Use of 3 unknown easyconfig parameters detected in > > > gompi-2018b.eb: binutilsver, gccver, tcver > > > If these are just local variables please rename them to start with > > > 'local_', or try using --fix-deprecated-easyconfigs to do this > > > automatically. > > > For more information, see > > > > > > https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html > > > > > . > > > > > > > > > WARNING: Deprecated functionality, will no longer work in v5.0: > > Use of > > > 'dummy' toolchain is deprecated, use 'system' toolchain instead; > see > > > > > > http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html > > > > > for more information > > > > > > > > > WARNING: Use of 3 unknown easyconfig parameters detected in > > > gompi-2018b.eb: binutilsver, gccver, tcver > > > If these are just local variables please rename them to start with > > > 'local_', or try using --fix-deprecated-easyconfigs to do this > > > automatically. > > > For more information, see > > > > > > https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html > > > > > . > > > > > > > > > WARNING: Use of 2 unknown easyconfig parameters detected in > > > GCC-4.9.3-binutils-2.25.eb: binutilsver, mpfr_version > > > If these are just local variables please rename them to start with > > > 'local_', or try using --fix-deprecated-easyconfigs to do this > > > automatically. > > > For more information, see > > > > > > https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html > > > > > . > > > > > > > > > WARNING: Deprecated functionality, will no longer work in v5.0: > > Use of > > > 'dummy' toolchain is deprecated, use 'system' toolchain instead; > see > > > > > > http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html > > > > > for more information > > > > > > > > > WARNING: Use of 2 unknown easyconfig parameters detected in > > > GCC-4.9.3-binutils-2.25.eb: binutilsver, mpfr_version > > > If these are just local variables please rename them to start with > > > 'local_', or try using --fix-deprecated-easyconfigs to do this > > > automatically. > > > For more information, see > > > > > > https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html > > > > > . > > > > > > > > > WARNING: Use of 2 unknown easyconfig parameters detected in > > > GNU-4.9.3-2.25.eb: binutilsver, gccver > > > If these are just local variables please rename them to start with > > > 'local_', or try using --fix-deprecated-easyconfigs to do this > > > automatically. > > > For more information, see > > > > > > https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html > > > > > . > > > > > > > > > WARNING: Use of 3 unknown easyconfig parameters detected in > > > OpenMPI-1.8.8-GNU-4.9.3-2.25.eb: binfile, libfile, libs > > > If these are just local variables please rename them to start with > > > 'local_', or try using --fix-deprecated-easyconfigs to do this > > > automatically. > > > For more information, see > > > > > > https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html > > > > > . > > > > > > > > > WARNING: Use of 2 unknown easyconfig parameters detected in > > > GNU-4.9.3-2.25.eb: binutilsver, gccver > > > If these are just local variables please rename them to start with > > > 'local_', or try using --fix-deprecated-easyconfigs to do this > > > automatically. > > > For more information, see > > > > > > https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html > > > > > . > > > > > > ERROR: Failed to process easyconfig > > > > > > > /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/s/ScaLAPACK/ScaLAPACK-2.0.2-gompi-2018b-OpenBLAS-0.3.1.eb: > > > > > Failed to determine minimal toolchain for dep OpenBLAS > > 0.3.1-GCC-7.3.0-2.30 > > > > > > On Mon, Nov 18, 2019 at 4:28 PM Kenneth Hoste > > <kenneth.ho...@ugent.be <mailto:kenneth.ho...@ugent.be> > > > <mailto:kenneth.ho...@ugent.be <mailto:kenneth.ho...@ugent.be>>> > > wrote: > > > > > > On 18/11/2019 21:50, Niek de Klein wrote: > > > > Hi Kenneth, > > > > > > > > I spoke too soon, some of my installations work but not > > all. When > > > I try > > > > to isntall Autoconf-2.69-foss-2018.eb I get > > > > > > > > ERROR: Failed to process easyconfig > > > > > > > > > > > /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/a/Autoconf/Autoconf-2.69-foss-2018b.eb: > > > > > > > Failed to determine minimal toolchain for dep M4 1.4.17 > > > > > > > > with eb --search I get: > > > > > > > > eb --search M4-1.4.17-foss-2018b > > > > * > > > > > > > > > > > /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.17-foss-2018b.eb > > > > > > > > eb --search ^foss-2018b.eb > > > > * > > > > > > > > > > > /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/f/foss/foss-2018b.eb > > > > > > > > So it can find both the M4-1.4.17-foss-2018b.eb file and > the > > > > foss-2018.eb file. > > > > > > Can you try "eb M4-1.4.17-foss-2018b.eb -D"? > > > > > > > > > > > > > > Kind regards, > > > > Niek > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 18, 2019 at 3:21 PM Niek de Klein > > > <niekdekl...@gmail.com <mailto:niekdekl...@gmail.com> > > <mailto:niekdekl...@gmail.com <mailto:niekdekl...@gmail.com>> > > > > <mailto:niekdekl...@gmail.com > > <mailto:niekdekl...@gmail.com> <mailto:niekdekl...@gmail.com > > <mailto:niekdekl...@gmail.com>>>> wrote: > > > > > > > > Hi Kenneth, > > > > > > > > Thanks for your help! > > > > > > > > 1. is solved now with your suggestion of > > > > EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1 > > > > > > > > Now that the bootstrap.py ran correctly it can also > > find the > > > > easybuild files in the location that I have > > configured, so this > > > > solves 1 and 2 as well. > > > > > > > > Thanks, > > > > Niek > > > > > > > > On Mon, Nov 18, 2019 at 3:01 PM Kenneth Hoste > > > > <kenneth.ho...@ugent.be > > <mailto:kenneth.ho...@ugent.be> <mailto:kenneth.ho...@ugent.be > > <mailto:kenneth.ho...@ugent.be>> > > > <mailto:kenneth.ho...@ugent.be > > <mailto:kenneth.ho...@ugent.be> <mailto:kenneth.ho...@ugent.be > > <mailto:kenneth.ho...@ugent.be>>>> wrote: > > > > > > > > Dear Niek, > > > > > > > > Sorry that you're hitting these issues, let's try > > and get > > > them > > > > sorted... > > > > > > > > On 18/11/2019 20:00, Niek de Klein wrote: > > > > > Hi all, > > > > > > > > > > I am trying to have my own installation of > > EasyBuild, > > > but I > > > > have 3 problems: > > > > > > > > > > 1. When I try to run bootstrap.py it gives an > > error that > > > > md5sums are > > > > > missing, but I'm not sure where to add them as I > > > thought that > > > > should > > > > > then have been included in the > > EasyBuild-4.0.1.eb file: > > > > > > > > > > easybuild.tools.build_log.EasyBuildError: > > "Build of > > > > > /tmp/tmpfcUSAH/EasyBuild-4.0.1.eb failed (err: > > 'build > > > failed > > > > (first 300 > > > > > chars): Missing checksum for > > > easybuild-framework-4.0.1.tar.gz')" > > > > > > > > Sorry for not responding to the GitHub issue [1] > you > > > opened on this > > > > (will do so now with the answer below). > > > > > > > > We still need to clean up the bootstrap script a > bit > > > after the > > > > EasyBuild > > > > v4.0 release, it's still doing a bunch of stuff it > > shouldn't > > > > anymore, > > > > and it's not compatible yet with Python 3 (see also > > > > > > https://github.com/easybuilders/easybuild-framework/pull/3028). > > > > > > > > > > > > Your active EasyBuild configuration has > > --enforce-checksums > > > > enabled (via > > > > $EASYBUILD_ENFORCE_CHECKSUMS), which explains the > > problem > > > you're > > > > seeing. > > > > > > > > The bootstrap script auto-generates an easyconfig > > file, > > > but it > > > > doesn't > > > > include checksums (it could I guess, with a bit > > more work). > > > > > > > > You can dance around this though, by temporarily > > > disabling the > > > > checksums > > > > requirement, only for the bootstrap: > > > > > > > > EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1 python > > bootstrap <path> > > > > > > > > > > > > If you want to, you can bootstrap EasyBuild to a > > temporary > > > > location (for > > > > example /tmp/$USER), and then use that > installation to > > > pull in an > > > > easyconfig file for EasyBuild v4.0.1 that does have > > > checksums, > > > > using "eb > > > > --from-pr 9149 --installpath <final_path>" (which > > pulls > > > in the > > > > easyconfig from > > > > > > > > > > https://github.com/easybuilders/easybuild-easyconfigs/pull/9149/files) > > > > > > > > for the actual installation you intend to use. > > > > > > > > > > > > > > > > > > > 2. I changed the robo and installation path to > be > > > different, > > > > but I keep > > > > > getting "Failed to determine minimal toolchain > > for dep > > > ..." > > > > errors. The > > > > > eb --show-config output is at the end of this > > e-mail, and > > > > shows that my > > > > > robot-paths is > > > > > > > > > > > > > > > /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/, > > > > > > > > > which is where I cloned the easyconfigs > repository. > > > > > > > > What happens if you use "eb --search" with this > setup, > > > can 'eb' > > > > find > > > > easyconfig files? > > > > > > > > Try something like "eb --search tensorflow-1.14" > > for example. > > > > > > > > > > > > > > > > > > 3. Even if the robo path wasn't working, tools > > that are > > > > already in my > > > > > module avail are not being picked up. For > example, > > > EasyBuild > > > > that is not > > > > > working uses "toolchain = {'name': 'foss', > > 'version': > > > > '2018b'}" and has > > > > > as one of the dependencies "('ncurses', > > '6.0'),". When > > > I try > > > > to install > > > > > it I get > > > > > > > > > > ERROR: Failed to process easyconfig > > > > > > > > > > > > > > > > /groups/umcg-biogen/tmp01/tools/brain_eQTL/EasyBuild/easyconfigs/SAMtools-1.5-foss-2018b.eb: > > > > > > > > > Failed to determine minimal toolchain for dep > > ncurses 6.0 > > > > > > > > > > but when I do ml avail ncur I get: > > > > > > > > > > > > ------------------------------------------------------ > > > > > /groups/umcg-biogen/tmp01/apps/modules/devel > > > > > > > ------------------------------------------------------- > > > > > ncurses/6.0-foss-2018b > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > /apps/modules/devel > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > ncurses/6.0 ncurses/6.1-foss-2018b > > > > ncurses/6.1-GCCcore-7.3.0 (D) > > > > > > > > > > So the ncurses module is already available > > > > > > > > > > > > > > > Am I missing something in my configuration? > > > > > > > > This is a consequence of having > --minimal-toolchains > > > enabled, which > > > > requires that EasyBuild is able to find the > easyconfig > > > file for the > > > > toolchain (foss-2018b.eb in this case). > > > > > > > > It needs to figure out what the (versions of) the > > > subtoolchains > > > > are for > > > > foss/2018b, so it can check for ncurses installed > with > > > GCCcore > > > > first, > > > > before gompi, before foss. > > > > > > > > So this is basically the same issue as with > > problem 2, I > > > bet "eb > > > > --search ^foss-2018b.eb" doesn't give you any > result? > > > > > > > > > > > > Are you 100% sure the path you specified for > > --robot-paths is > > > > correct? > > > > > > > > Note that you can also prepend or append to the > robot > > > search path > > > > (rather than override the default), see > > > > > > > > > > https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path > > > > > > > > . > > > > > > > > I hope this helps... > > > > > > > > > > > > regards, > > > > > > > > Kenneth > > > > > > > > > > > > [1] > > > https://github.com/easybuilders/easybuild-framework/issues/3079 > > > > > > > > > > > > > > Thanks for your help! > > > > > Niek de Klein > > > > > > > > > > > > > > > # > > > > > # Current EasyBuild configuration > > > > > # (C: command line argument, D: default value, > E: > > > environment > > > > variable, > > > > > F: configuration file) > > > > > # > > > > > buildpath (E) = > > > > > > > /groups/umcg-biogen/tmp01/apps//.tmp/easybuild/builds/ > > > > > containerpath (D) = > > > > > /home/umcg-ndeklein/.local/easybuild/containers > > > > > enforce-checksums (E) = True > > > > > installpath (E) = > > > /groups/umcg-biogen/tmp01/apps/ > > > > > minimal-toolchains (E) = True > > > > > repositorypath (D) = > > > > > > /home/umcg-ndeklein/.local/easybuild/ebfiles_repo > > > > > robot-paths (E) = > > > > > > > > > > > > > > > /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/ > > > > > set-gid-bit (E) = True > > > > > sourcepath (E) = > > > > /groups/umcg-biogen/tmp01/apps//sources/ > > > > > umask (E) = 002 > > > > > verify-easyconfig-filenames (E) = True > > > > > > > > > > > > > > > > > > > >
