Re: [easybuild] Force rebuild of module plus all dependencies
"Alan O'Cais" writes: > EasyBuild recognises modules only, if the module name is available then it is > installed. In your case you have the path to the original module tree in > MODULEPATH so it is able to load the exact module. Try > unset MODULEPATH > and add the --missing option to see what would get installed (the default > location is added by EasyBuild to MODULEPATH, so it can be completely unset > when you call the command). Thanks Alan and Markus. I was staring at the EasyBuild configuration and forgetting about the Lmod configuration. Now I can get my sword out ... (https://xkcd.com/303/) Cheers, Loris > On Fri, 2 Dec 2022 at 11:58, Loris Bennett wrote: > > "Alan O'Cais" writes: > > > Ever configuartion setting can be done in multiple ways, see > https://easybuilders.github.io/easybuild-tutorial/2022-isc22/configuration/ > for a > good > > overview. > > > > The easiest way to do arch support is to have use .../easybuild/ as > the --prefix for each node type. The only stuff you should really worry > about > > have multiple copies of are the sources, so you can set --sourcepath to > .../easybuild/sources to avoid that duplication. For each node then you > > configure the profile so that it detects the right path(s) for `module use > /path/to//modules/all` > > OK > > > On Fri, 2 Dec 2022 at 11:01, Loris Bennett > wrote: > > > > I was basing my approach on the first answer given here: > > > > > https://www.reddit.com/r/HPC/comments/y1gdch/how_to_manage_easybuild_on_hpc_with_multiple_cpu/ > > > > But I see now that the problem being solved there is slightly different > > to mine, as two existing clusters were being merged, so the software for > > the two different architectures had already been built. > > > > "Alan O'Cais" writes: > > > > > Sorry, wrong name: --installpath-modules > > > > OK, thanks. Is there no corresponding environment variable? > > > > > If you are not using this explicitly in your config, then you are > relying on EASYBUILD_PREFIX to set the default, see > > > > https://docs.easybuild.io/en/latest/Configuration.html#mandatory-configuration-settings > for details. By setting _just_ EASYBUILD_INSTALLPATH, > you > > are > > > only changing where software gets installed, everything else is > remaining the same (and EasyBuild relies only on the module files to know if > > something > > > is installed or not). > > Something seems to be wrong with my set-up. I have defined my own variable > >[build@a001 ~]$ echo $EASYBUILD_INSTALLPATH_MODULES >/trinity/shared/easybuild/arch/amd/zen3/modules > > which in fact EasyBuild seems to recognise: > >[build@a001 ~]$ eb --show-config ># ># Current EasyBuild configuration ># (C: command line argument, D: default value, E: environment variable, F: > configuration file) ># >buildpath (F) = /trinity/shared/easybuild/build >containerpath (F) = /trinity/shared/easybuild/containers >hooks (F) = /trinity/home/build/hooks/hpc_hooks.py >installpath (E) = /trinity/shared/easybuild/arch/amd/zen3/software >installpath-modules (E) = /trinity/shared/easybuild/arch/amd/zen3/modules >packagepath (F) = /trinity/shared/easybuild/packages >prefix (E) = /trinity/shared/easybuild >repositorypath (F) = /trinity/shared/easybuild/ebfiles_repo >robot-paths (D) = > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs >sourcepath (F) = /trinity/shared/easybuild/sources > > However, EasyBuild still doesn't want to build the dependency: > > [build@a001 ~]$ eb Bison-3.8.2.eb > --installpath-modules=$EASYBUILD_INSTALLPATH_MODULES --robot --dry-run --force > == Temporary log file in case of crash > /tmp/eb-aqpm3paq/easybuild-vqgytl2y.log > == found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so > using it... > == found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so > using it... > Dry run: printing build status of easyconfigs and dependencies > * [x] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb > (module: M4/1.4.19) > * [F] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb > (module: Bison/3.8.2) > == Temporary log file(s) /tmp/eb-aqpm3paq/easybuild-vqgytl2y.log* have been > removed. > == Temporary directory /tmp/eb-aqpm3paq has been removed. > > What am I doing wrong? > > Cheers, > > Loris > > [snip (116 lines)] > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin > -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin
Re: [easybuild] Force rebuild of module plus all dependencies
EasyBuild recognises modules only, if the module name is available then it is installed. In your case you have the path to the original module tree in MODULEPATH so it is able to load the exact module. Try unset MODULEPATH and add the --missing option to see what would get installed (the default location is added by EasyBuild to MODULEPATH, so it can be completely unset when you call the command). On Fri, 2 Dec 2022 at 11:58, Loris Bennett wrote: > "Alan O'Cais" writes: > > > Ever configuartion setting can be done in multiple ways, see > https://easybuilders.github.io/easybuild-tutorial/2022-isc22/configuration/ > for a good > > overview. > > > > The easiest way to do arch support is to have use .../easybuild/ > as the --prefix for each node type. The only stuff you should really worry > about > > have multiple copies of are the sources, so you can set --sourcepath to > .../easybuild/sources to avoid that duplication. For each node then you > > configure the profile so that it detects the right path(s) for `module > use /path/to//modules/all` > > OK > > > On Fri, 2 Dec 2022 at 11:01, Loris Bennett > wrote: > > > > I was basing my approach on the first answer given here: > > > > > https://www.reddit.com/r/HPC/comments/y1gdch/how_to_manage_easybuild_on_hpc_with_multiple_cpu/ > > > > But I see now that the problem being solved there is slightly different > > to mine, as two existing clusters were being merged, so the software for > > the two different architectures had already been built. > > > > "Alan O'Cais" writes: > > > > > Sorry, wrong name: --installpath-modules > > > > OK, thanks. Is there no corresponding environment variable? > > > > > If you are not using this explicitly in your config, then you are > relying on EASYBUILD_PREFIX to set the default, see > > > > https://docs.easybuild.io/en/latest/Configuration.html#mandatory-configuration-settings > for details. By setting _just_ EASYBUILD_INSTALLPATH, you > > are > > > only changing where software gets installed, everything else is > remaining the same (and EasyBuild relies only on the module files to know if > > something > > > is installed or not). > > Something seems to be wrong with my set-up. I have defined my own variable > > [build@a001 ~]$ echo $EASYBUILD_INSTALLPATH_MODULES > /trinity/shared/easybuild/arch/amd/zen3/modules > > which in fact EasyBuild seems to recognise: > > [build@a001 ~]$ eb --show-config > # > # Current EasyBuild configuration > # (C: command line argument, D: default value, E: environment variable, > F: configuration file) > # > buildpath (F) = /trinity/shared/easybuild/build > containerpath (F) = /trinity/shared/easybuild/containers > hooks (F) = /trinity/home/build/hooks/hpc_hooks.py > installpath (E) = > /trinity/shared/easybuild/arch/amd/zen3/software > installpath-modules (E) = /trinity/shared/easybuild/arch/amd/zen3/modules > packagepath (F) = /trinity/shared/easybuild/packages > prefix (E) = /trinity/shared/easybuild > repositorypath (F) = /trinity/shared/easybuild/ebfiles_repo > robot-paths (D) = > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs > sourcepath (F) = /trinity/shared/easybuild/sources > > However, EasyBuild still doesn't want to build the dependency: > > [build@a001 ~]$ eb Bison-3.8.2.eb > --installpath-modules=$EASYBUILD_INSTALLPATH_MODULES --robot --dry-run > --force > == Temporary log file in case of crash > /tmp/eb-aqpm3paq/easybuild-vqgytl2y.log > == found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, > so using it... > == found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, > so using it... > Dry run: printing build status of easyconfigs and dependencies > * [x] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb > (module: M4/1.4.19) > * [F] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb > (module: Bison/3.8.2) > == Temporary log file(s) /tmp/eb-aqpm3paq/easybuild-vqgytl2y.log* have > been removed. > == Temporary directory /tmp/eb-aqpm3paq has been removed. > > What am I doing wrong? > > Cheers, > > Loris > > [snip (116 lines)] > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin >
Re: [easybuild] Force rebuild of module plus all dependencies
Loris, Do you have $MODULEPATH also set to the new modules tree? The setting `--installpath-modules` only specifies where *new* modules should go, but AFAIK doesn't fully override the search paths for existing modules. Markus On 12/2/22 11:58, Loris Bennett wrote: "Alan O'Cais" writes: Ever configuartion setting can be done in multiple ways, see https://easybuilders.github.io/easybuild-tutorial/2022-isc22/configuration/ for a good overview. The easiest way to do arch support is to have use .../easybuild/ as the --prefix for each node type. The only stuff you should really worry about have multiple copies of are the sources, so you can set --sourcepath to .../easybuild/sources to avoid that duplication. For each node then you configure the profile so that it detects the right path(s) for `module use /path/to//modules/all` OK On Fri, 2 Dec 2022 at 11:01, Loris Bennett wrote: I was basing my approach on the first answer given here: https://www.reddit.com/r/HPC/comments/y1gdch/how_to_manage_easybuild_on_hpc_with_multiple_cpu/ But I see now that the problem being solved there is slightly different to mine, as two existing clusters were being merged, so the software for the two different architectures had already been built. "Alan O'Cais" writes: > Sorry, wrong name: --installpath-modules OK, thanks. Is there no corresponding environment variable? > If you are not using this explicitly in your config, then you are relying on EASYBUILD_PREFIX to set the default, see > https://docs.easybuild.io/en/latest/Configuration.html#mandatory-configuration-settings for details. By setting _just_ EASYBUILD_INSTALLPATH, you are > only changing where software gets installed, everything else is remaining the same (and EasyBuild relies only on the module files to know if something > is installed or not). Something seems to be wrong with my set-up. I have defined my own variable [build@a001 ~]$ echo $EASYBUILD_INSTALLPATH_MODULES /trinity/shared/easybuild/arch/amd/zen3/modules which in fact EasyBuild seems to recognise: [build@a001 ~]$ eb --show-config # # Current EasyBuild configuration # (C: command line argument, D: default value, E: environment variable, F: configuration file) # buildpath (F) = /trinity/shared/easybuild/build containerpath (F) = /trinity/shared/easybuild/containers hooks (F) = /trinity/home/build/hooks/hpc_hooks.py installpath (E) = /trinity/shared/easybuild/arch/amd/zen3/software installpath-modules (E) = /trinity/shared/easybuild/arch/amd/zen3/modules packagepath (F) = /trinity/shared/easybuild/packages prefix (E) = /trinity/shared/easybuild repositorypath (F) = /trinity/shared/easybuild/ebfiles_repo robot-paths (D) = /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs sourcepath (F) = /trinity/shared/easybuild/sources However, EasyBuild still doesn't want to build the dependency: [build@a001 ~]$ eb Bison-3.8.2.eb --installpath-modules=$EASYBUILD_INSTALLPATH_MODULES --robot --dry-run --force == Temporary log file in case of crash /tmp/eb-aqpm3paq/easybuild-vqgytl2y.log == found valid index for /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so using it... == found valid index for /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so using it... Dry run: printing build status of easyconfigs and dependencies * [x] /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb (module: M4/1.4.19) * [F] /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb (module: Bison/3.8.2) == Temporary log file(s) /tmp/eb-aqpm3paq/easybuild-vqgytl2y.log* have been removed. == Temporary directory /tmp/eb-aqpm3paq has been removed. What am I doing wrong? Cheers, Loris [snip (116 lines)] -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 Email: m.gei...@fz-juelich.de WWW: https://www.fz-juelich.de/ias/jsc - - Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior - -
Re: [easybuild] Force rebuild of module plus all dependencies
"Alan O'Cais" writes: > Ever configuartion setting can be done in multiple ways, see > https://easybuilders.github.io/easybuild-tutorial/2022-isc22/configuration/ > for a good > overview. > > The easiest way to do arch support is to have use .../easybuild/ as the > --prefix for each node type. The only stuff you should really worry about > have multiple copies of are the sources, so you can set --sourcepath to > .../easybuild/sources to avoid that duplication. For each node then you > configure the profile so that it detects the right path(s) for `module use > /path/to//modules/all` OK > On Fri, 2 Dec 2022 at 11:01, Loris Bennett wrote: > > I was basing my approach on the first answer given here: > > > https://www.reddit.com/r/HPC/comments/y1gdch/how_to_manage_easybuild_on_hpc_with_multiple_cpu/ > > But I see now that the problem being solved there is slightly different > to mine, as two existing clusters were being merged, so the software for > the two different architectures had already been built. > > "Alan O'Cais" writes: > > > Sorry, wrong name: --installpath-modules > > OK, thanks. Is there no corresponding environment variable? > > > If you are not using this explicitly in your config, then you are relying > on EASYBUILD_PREFIX to set the default, see > > > https://docs.easybuild.io/en/latest/Configuration.html#mandatory-configuration-settings > for details. By setting _just_ EASYBUILD_INSTALLPATH, you > are > > only changing where software gets installed, everything else is remaining > the same (and EasyBuild relies only on the module files to know if > something > > is installed or not). Something seems to be wrong with my set-up. I have defined my own variable [build@a001 ~]$ echo $EASYBUILD_INSTALLPATH_MODULES /trinity/shared/easybuild/arch/amd/zen3/modules which in fact EasyBuild seems to recognise: [build@a001 ~]$ eb --show-config # # Current EasyBuild configuration # (C: command line argument, D: default value, E: environment variable, F: configuration file) # buildpath (F) = /trinity/shared/easybuild/build containerpath (F) = /trinity/shared/easybuild/containers hooks (F) = /trinity/home/build/hooks/hpc_hooks.py installpath (E) = /trinity/shared/easybuild/arch/amd/zen3/software installpath-modules (E) = /trinity/shared/easybuild/arch/amd/zen3/modules packagepath (F) = /trinity/shared/easybuild/packages prefix (E) = /trinity/shared/easybuild repositorypath (F) = /trinity/shared/easybuild/ebfiles_repo robot-paths (D) = /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs sourcepath (F) = /trinity/shared/easybuild/sources However, EasyBuild still doesn't want to build the dependency: [build@a001 ~]$ eb Bison-3.8.2.eb --installpath-modules=$EASYBUILD_INSTALLPATH_MODULES --robot --dry-run --force == Temporary log file in case of crash /tmp/eb-aqpm3paq/easybuild-vqgytl2y.log == found valid index for /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so using it... == found valid index for /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so using it... Dry run: printing build status of easyconfigs and dependencies * [x] /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb (module: M4/1.4.19) * [F] /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb (module: Bison/3.8.2) == Temporary log file(s) /tmp/eb-aqpm3paq/easybuild-vqgytl2y.log* have been removed. == Temporary directory /tmp/eb-aqpm3paq has been removed. What am I doing wrong? Cheers, Loris [snip (116 lines)] -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin
Re: [easybuild] Force rebuild of module plus all dependencies
Ever configuartion setting can be done in multiple ways, see https://easybuilders.github.io/easybuild-tutorial/2022-isc22/configuration/ for a good overview. The easiest way to do arch support is to have use .../easybuild/ as the --prefix for each node type. The only stuff you should really worry about have multiple copies of are the sources, so you can set --sourcepath to .../easybuild/sources to avoid that duplication. For each node then you configure the profile so that it detects the right path(s) for `module use /path/to//modules/all` On Fri, 2 Dec 2022 at 11:01, Loris Bennett wrote: > > I was basing my approach on the first answer given here: > > > https://www.reddit.com/r/HPC/comments/y1gdch/how_to_manage_easybuild_on_hpc_with_multiple_cpu/ > > But I see now that the problem being solved there is slightly different > to mine, as two existing clusters were being merged, so the software for > the two different architectures had already been built. > > "Alan O'Cais" writes: > > > Sorry, wrong name: --installpath-modules > > OK, thanks. Is there no corresponding environment variable? > > > If you are not using this explicitly in your config, then you are > relying on EASYBUILD_PREFIX to set the default, see > > > https://docs.easybuild.io/en/latest/Configuration.html#mandatory-configuration-settings > for details. By setting _just_ EASYBUILD_INSTALLPATH, you are > > only changing where software gets installed, everything else is > remaining the same (and EasyBuild relies only on the module files to know > if something > > is installed or not). > > Ah, OK. Is there a recommend directory layout? I can think of > > easybuild/ > |-- build > |-- ebfiles_repo > |-- moduleData > |-- modules > | `-- arch > | |-- intel > | `-- amd > |-- software > | `-- arch > | |-- intel > | `-- amd > `-- sources > > or > > easybuild/ > |-- build > |-- ebfiles_repo > |-- moduleData > |-- arch > | |-- intel > | | |-- software > | | `-- modules > | `-- amd > | | |-- software > | | `-- modules > `-- sources > > Are there any factors which would make one preferable to the other? > > > On Fri, 2 Dec 2022 at 09:55, Loris Bennett > wrote: > > > > Hi Alan, > > > > Is EASYBUILD_MODULEPATH documented anywhere? It is not set in my > > environment and my search on the web didn't turn up any results. > > > > Cheers, > > > > Loris > > > > "Alan O'Cais" writes: > > > > > You are changing the EASYBUILD_INSTALLPATH, but not changing > EASYBUILD_MODULEPATH. This means you are overwriting the existing module > tree. You > > don't > > > want this, instead you build a new module tree somewhere else. Since > the tree will be empty, EasyBuild will also install all the deps. > > > > > > On Fri, 2 Dec 2022 at 09:27, Loris Bennett < > loris.benn...@fu-berlin.de> wrote: > > > > > > Hi Ole, > > > > > > Ole Holm Nielsen writes: > > > > > > > Hi Loris, > > > > > > > > On 12/2/22 08:27, Loris Bennett wrote: > > > >> How do I force a total rebuild of, say, a foss toolchain for a > different > > > >> CPU architecture? > > > >> Up to now I had a homogeneous cluster with Intel Xeon CPUs, now we > > > >> have > > > >> acquired some nodes with AMD Epyc CPUs for which I need to build > > > >> software. > > > >> I have modified EASYBUILD_INSTALLPATH to point to directory for > the > > > >> new > > > >> architecture and prepended a corresponding directory for the > modules to > > > >> MODULEPATH. However, running EasyBuild with the option --force > just > > > >> rebuilds the package specified, not the dependencies. > > > >> What is the correct way to go about this? > > > > > > > > Probably there are multiple ways to set up modules for multiple > > > > architectures :-) My choice was to create completely different > module > > > > trees for each type of hardware (we have 4 generations of Intel > Xeon). > > > > My notes are in this Wiki page: > > > > > https://wiki.fysik.dtu.dk/Niflheim_system/EasyBuild_modules/#automounting-the-cpu-architecture-dependent-modules-directory > > > > > > > > IHTH /Ole > > > > > > Thanks for the link. That looks like an good way of dealing with > > > switching between different architectures. > > > > > > However, I am not yet quite that far. My question was more to do > with > > > how to actually rebuild the software in a second branch. If I change > > > EASYBUILD_INSTALLPATH and MODULEPATH as described abouve and then run > > > EasyBuild with force I get, say, the following: > > > > > >[build@a001 ~]$ eb Bison-3.8.2.eb --robot --force --dry-run > > >== Temporary log file in case of crash > /tmp/eb-akrn565i/easybuild-0vsmv95q.log > > >== found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, > so using it... > > >== found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6
Re: [easybuild] Force rebuild of module plus all dependencies
I was basing my approach on the first answer given here: https://www.reddit.com/r/HPC/comments/y1gdch/how_to_manage_easybuild_on_hpc_with_multiple_cpu/ But I see now that the problem being solved there is slightly different to mine, as two existing clusters were being merged, so the software for the two different architectures had already been built. "Alan O'Cais" writes: > Sorry, wrong name: --installpath-modules OK, thanks. Is there no corresponding environment variable? > If you are not using this explicitly in your config, then you are relying on > EASYBUILD_PREFIX to set the default, see > https://docs.easybuild.io/en/latest/Configuration.html#mandatory-configuration-settings > for details. By setting _just_ EASYBUILD_INSTALLPATH, you are > only changing where software gets installed, everything else is remaining the > same (and EasyBuild relies only on the module files to know if something > is installed or not). Ah, OK. Is there a recommend directory layout? I can think of easybuild/ |-- build |-- ebfiles_repo |-- moduleData |-- modules | `-- arch | |-- intel | `-- amd |-- software | `-- arch | |-- intel | `-- amd `-- sources or easybuild/ |-- build |-- ebfiles_repo |-- moduleData |-- arch | |-- intel | | |-- software | | `-- modules | `-- amd | | |-- software | | `-- modules `-- sources Are there any factors which would make one preferable to the other? > On Fri, 2 Dec 2022 at 09:55, Loris Bennett wrote: > > Hi Alan, > > Is EASYBUILD_MODULEPATH documented anywhere? It is not set in my > environment and my search on the web didn't turn up any results. > > Cheers, > > Loris > > "Alan O'Cais" writes: > > > You are changing the EASYBUILD_INSTALLPATH, but not changing > EASYBUILD_MODULEPATH. This means you are overwriting the existing module > tree. You > don't > > want this, instead you build a new module tree somewhere else. Since the > tree will be empty, EasyBuild will also install all the deps. > > > > On Fri, 2 Dec 2022 at 09:27, Loris Bennett > wrote: > > > > Hi Ole, > > > > Ole Holm Nielsen writes: > > > > > Hi Loris, > > > > > > On 12/2/22 08:27, Loris Bennett wrote: > > >> How do I force a total rebuild of, say, a foss toolchain for a > different > > >> CPU architecture? > > >> Up to now I had a homogeneous cluster with Intel Xeon CPUs, now we > > >> have > > >> acquired some nodes with AMD Epyc CPUs for which I need to build > > >> software. > > >> I have modified EASYBUILD_INSTALLPATH to point to directory for the > > >> new > > >> architecture and prepended a corresponding directory for the modules to > > >> MODULEPATH. However, running EasyBuild with the option --force just > > >> rebuilds the package specified, not the dependencies. > > >> What is the correct way to go about this? > > > > > > Probably there are multiple ways to set up modules for multiple > > > architectures :-) My choice was to create completely different module > > > trees for each type of hardware (we have 4 generations of Intel Xeon). > > > My notes are in this Wiki page: > > > > https://wiki.fysik.dtu.dk/Niflheim_system/EasyBuild_modules/#automounting-the-cpu-architecture-dependent-modules-directory > > > > > > IHTH /Ole > > > > Thanks for the link. That looks like an good way of dealing with > > switching between different architectures. > > > > However, I am not yet quite that far. My question was more to do with > > how to actually rebuild the software in a second branch. If I change > > EASYBUILD_INSTALLPATH and MODULEPATH as described abouve and then run > > EasyBuild with force I get, say, the following: > > > >[build@a001 ~]$ eb Bison-3.8.2.eb --robot --force --dry-run > >== Temporary log file in case of crash > /tmp/eb-akrn565i/easybuild-0vsmv95q.log > >== found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so > using it... > >== found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so > using it... > >Dry run: printing build status of easyconfigs and dependencies > > * [x] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb > (module: M4/1.4.19) > > * [F] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb > (module: Bison/3.8.2) > > > > How to I get EasyBuild to build M4 automatically as well? > > > > Cheers, > > > > Loris > > > > -- > > Dr. Loris Bennett (Herr/Mr) > > ZEDAT, Freie Universität Berlin > > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin > -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin
Re: [easybuild] Force rebuild of module plus all dependencies
Sorry, wrong name: --installpath-modules If you are not using this explicitly in your config, then you are relying on EASYBUILD_PREFIX to set the default, see https://docs.easybuild.io/en/latest/Configuration.html#mandatory-configuration-settings for details. By setting _just_ EASYBUILD_INSTALLPATH, you are only changing where software gets installed, everything else is remaining the same (and EasyBuild relies only on the module files to know if something is installed or not). On Fri, 2 Dec 2022 at 09:55, Loris Bennett wrote: > Hi Alan, > > Is EASYBUILD_MODULEPATH documented anywhere? It is not set in my > environment and my search on the web didn't turn up any results. > > Cheers, > > Loris > > "Alan O'Cais" writes: > > > You are changing the EASYBUILD_INSTALLPATH, but not changing > EASYBUILD_MODULEPATH. This means you are overwriting the existing module > tree. You don't > > want this, instead you build a new module tree somewhere else. Since the > tree will be empty, EasyBuild will also install all the deps. > > > > On Fri, 2 Dec 2022 at 09:27, Loris Bennett > wrote: > > > > Hi Ole, > > > > Ole Holm Nielsen writes: > > > > > Hi Loris, > > > > > > On 12/2/22 08:27, Loris Bennett wrote: > > >> How do I force a total rebuild of, say, a foss toolchain for a > different > > >> CPU architecture? > > >> Up to now I had a homogeneous cluster with Intel Xeon CPUs, now we > > >> have > > >> acquired some nodes with AMD Epyc CPUs for which I need to build > > >> software. > > >> I have modified EASYBUILD_INSTALLPATH to point to directory for the > > >> new > > >> architecture and prepended a corresponding directory for the modules > to > > >> MODULEPATH. However, running EasyBuild with the option --force just > > >> rebuilds the package specified, not the dependencies. > > >> What is the correct way to go about this? > > > > > > Probably there are multiple ways to set up modules for multiple > > > architectures :-) My choice was to create completely different > module > > > trees for each type of hardware (we have 4 generations of Intel Xeon). > > > My notes are in this Wiki page: > > > > https://wiki.fysik.dtu.dk/Niflheim_system/EasyBuild_modules/#automounting-the-cpu-architecture-dependent-modules-directory > > > > > > IHTH /Ole > > > > Thanks for the link. That looks like an good way of dealing with > > switching between different architectures. > > > > However, I am not yet quite that far. My question was more to do with > > how to actually rebuild the software in a second branch. If I change > > EASYBUILD_INSTALLPATH and MODULEPATH as described abouve and then run > > EasyBuild with force I get, say, the following: > > > >[build@a001 ~]$ eb Bison-3.8.2.eb --robot --force --dry-run > >== Temporary log file in case of crash > /tmp/eb-akrn565i/easybuild-0vsmv95q.log > >== found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, > so using it... > >== found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, > so using it... > >Dry run: printing build status of easyconfigs and dependencies > > * [x] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb > (module: M4/1.4.19) > > * [F] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb > (module: Bison/3.8.2) > > > > How to I get EasyBuild to build M4 automatically as well? > > > > Cheers, > > > > Loris > > > > -- > > Dr. Loris Bennett (Herr/Mr) > > ZEDAT, Freie Universität Berlin > > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin >
Re: [easybuild] Force rebuild of module plus all dependencies
Hi Alan, Is EASYBUILD_MODULEPATH documented anywhere? It is not set in my environment and my search on the web didn't turn up any results. Cheers, Loris "Alan O'Cais" writes: > You are changing the EASYBUILD_INSTALLPATH, but not changing > EASYBUILD_MODULEPATH. This means you are overwriting the existing module > tree. You don't > want this, instead you build a new module tree somewhere else. Since the tree > will be empty, EasyBuild will also install all the deps. > > On Fri, 2 Dec 2022 at 09:27, Loris Bennett wrote: > > Hi Ole, > > Ole Holm Nielsen writes: > > > Hi Loris, > > > > On 12/2/22 08:27, Loris Bennett wrote: > >> How do I force a total rebuild of, say, a foss toolchain for a different > >> CPU architecture? > >> Up to now I had a homogeneous cluster with Intel Xeon CPUs, now we > >> have > >> acquired some nodes with AMD Epyc CPUs for which I need to build > >> software. > >> I have modified EASYBUILD_INSTALLPATH to point to directory for the > >> new > >> architecture and prepended a corresponding directory for the modules to > >> MODULEPATH. However, running EasyBuild with the option --force just > >> rebuilds the package specified, not the dependencies. > >> What is the correct way to go about this? > > > > Probably there are multiple ways to set up modules for multiple > > architectures :-) My choice was to create completely different module > > trees for each type of hardware (we have 4 generations of Intel Xeon). > > My notes are in this Wiki page: > > > https://wiki.fysik.dtu.dk/Niflheim_system/EasyBuild_modules/#automounting-the-cpu-architecture-dependent-modules-directory > > > > IHTH /Ole > > Thanks for the link. That looks like an good way of dealing with > switching between different architectures. > > However, I am not yet quite that far. My question was more to do with > how to actually rebuild the software in a second branch. If I change > EASYBUILD_INSTALLPATH and MODULEPATH as described abouve and then run > EasyBuild with force I get, say, the following: > >[build@a001 ~]$ eb Bison-3.8.2.eb --robot --force --dry-run >== Temporary log file in case of crash > /tmp/eb-akrn565i/easybuild-0vsmv95q.log >== found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so > using it... >== found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so > using it... >Dry run: printing build status of easyconfigs and dependencies > * [x] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb > (module: M4/1.4.19) > * [F] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb > (module: Bison/3.8.2) > > How to I get EasyBuild to build M4 automatically as well? > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin > -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin
Re: [easybuild] Force rebuild of module plus all dependencies
You are changing the EASYBUILD_INSTALLPATH, but not changing EASYBUILD_MODULEPATH. This means you are overwriting the existing module tree. You don't want this, instead you build a new module tree somewhere else. Since the tree will be empty, EasyBuild will also install all the deps. On Fri, 2 Dec 2022 at 09:27, Loris Bennett wrote: > Hi Ole, > > Ole Holm Nielsen writes: > > > Hi Loris, > > > > On 12/2/22 08:27, Loris Bennett wrote: > >> How do I force a total rebuild of, say, a foss toolchain for a different > >> CPU architecture? > >> Up to now I had a homogeneous cluster with Intel Xeon CPUs, now we > >> have > >> acquired some nodes with AMD Epyc CPUs for which I need to build > >> software. > >> I have modified EASYBUILD_INSTALLPATH to point to directory for the > >> new > >> architecture and prepended a corresponding directory for the modules to > >> MODULEPATH. However, running EasyBuild with the option --force just > >> rebuilds the package specified, not the dependencies. > >> What is the correct way to go about this? > > > > Probably there are multiple ways to set up modules for multiple > > architectures :-) My choice was to create completely different module > > trees for each type of hardware (we have 4 generations of Intel Xeon). > > My notes are in this Wiki page: > > > https://wiki.fysik.dtu.dk/Niflheim_system/EasyBuild_modules/#automounting-the-cpu-architecture-dependent-modules-directory > > > > IHTH /Ole > > Thanks for the link. That looks like an good way of dealing with > switching between different architectures. > > However, I am not yet quite that far. My question was more to do with > how to actually rebuild the software in a second branch. If I change > EASYBUILD_INSTALLPATH and MODULEPATH as described abouve and then run > EasyBuild with force I get, say, the following: > > [build@a001 ~]$ eb Bison-3.8.2.eb --robot --force --dry-run > == Temporary log file in case of crash > /tmp/eb-akrn565i/easybuild-0vsmv95q.log > == found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, > so using it... > == found valid index for > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, > so using it... > Dry run: printing build status of easyconfigs and dependencies >* [x] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb > (module: M4/1.4.19) >* [F] > /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb > (module: Bison/3.8.2) > > How to I get EasyBuild to build M4 automatically as well? > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin >
Re: [easybuild] Force rebuild of module plus all dependencies
Hi Ole, Ole Holm Nielsen writes: > Hi Loris, > > On 12/2/22 08:27, Loris Bennett wrote: >> How do I force a total rebuild of, say, a foss toolchain for a different >> CPU architecture? >> Up to now I had a homogeneous cluster with Intel Xeon CPUs, now we >> have >> acquired some nodes with AMD Epyc CPUs for which I need to build >> software. >> I have modified EASYBUILD_INSTALLPATH to point to directory for the >> new >> architecture and prepended a corresponding directory for the modules to >> MODULEPATH. However, running EasyBuild with the option --force just >> rebuilds the package specified, not the dependencies. >> What is the correct way to go about this? > > Probably there are multiple ways to set up modules for multiple > architectures :-) My choice was to create completely different module > trees for each type of hardware (we have 4 generations of Intel Xeon). > My notes are in this Wiki page: > https://wiki.fysik.dtu.dk/Niflheim_system/EasyBuild_modules/#automounting-the-cpu-architecture-dependent-modules-directory > > IHTH /Ole Thanks for the link. That looks like an good way of dealing with switching between different architectures. However, I am not yet quite that far. My question was more to do with how to actually rebuild the software in a second branch. If I change EASYBUILD_INSTALLPATH and MODULEPATH as described abouve and then run EasyBuild with force I get, say, the following: [build@a001 ~]$ eb Bison-3.8.2.eb --robot --force --dry-run == Temporary log file in case of crash /tmp/eb-akrn565i/easybuild-0vsmv95q.log == found valid index for /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so using it... == found valid index for /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs, so using it... Dry run: printing build status of easyconfigs and dependencies * [x] /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/m/M4/M4-1.4.19.eb (module: M4/1.4.19) * [F] /trinity/shared/easybuild/software/EasyBuild/4.6.2/easybuild/easyconfigs/b/Bison/Bison-3.8.2.eb (module: Bison/3.8.2) How to I get EasyBuild to build M4 automatically as well? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin
Re: [easybuild] Force rebuild of module plus all dependencies
Hi Loris, On 12/2/22 08:27, Loris Bennett wrote: How do I force a total rebuild of, say, a foss toolchain for a different CPU architecture? Up to now I had a homogeneous cluster with Intel Xeon CPUs, now we have acquired some nodes with AMD Epyc CPUs for which I need to build software. I have modified EASYBUILD_INSTALLPATH to point to directory for the new architecture and prepended a corresponding directory for the modules to MODULEPATH. However, running EasyBuild with the option --force just rebuilds the package specified, not the dependencies. What is the correct way to go about this? Probably there are multiple ways to set up modules for multiple architectures :-) My choice was to create completely different module trees for each type of hardware (we have 4 generations of Intel Xeon). My notes are in this Wiki page: https://wiki.fysik.dtu.dk/Niflheim_system/EasyBuild_modules/#automounting-the-cpu-architecture-dependent-modules-directory IHTH /Ole
[easybuild] Force rebuild of module plus all dependencies
Hi, How do I force a total rebuild of, say, a foss toolchain for a different CPU architecture? Up to now I had a homogeneous cluster with Intel Xeon CPUs, now we have acquired some nodes with AMD Epyc CPUs for which I need to build software. I have modified EASYBUILD_INSTALLPATH to point to directory for the new architecture and prepended a corresponding directory for the modules to MODULEPATH. However, running EasyBuild with the option --force just rebuilds the package specified, not the dependencies. What is the correct way to go about this? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin
