On Fri, 22 Jun 2018 at 18:26, <paye...@umd.edu<mailto:paye...@umd.edu>> wrote: I am investigating the use of easybuild when we roll out our next cluster.
One concern raised was from https://github.com/easybuilders/easybuild/wiki/Using-a-custom-module-naming-scheme where it clear states that different installation prefixes are needed for different easybuild naming schemes, or easybuild for schemeA might delete software built with schemeB. Actually that may be out of date. While what you describe is indeed the default behaviour, there is a specific setting that fixes the installdir naming scheme '--enable-fixed-installdir-naming-scheme' which makes the installation prefix independent of the module naming scheme. You can then regenerate the module tree for the a new naming scheme with '--module-only' (but this is known not to be perfect, some of the easyblocks make decisions at runtime based on things that happen in the other installation steps). I am still getting familiar with easybuild, but this suggests to me that if I plan to have a mix of software built with easybuild and software built manually and/or with some other software build tools, that they would need to be installed in distinct software trees. E.g., if I intend to build package foo with easybuild, putting it in /software/Core/foo/... and I intend to build package bar manually, then I better not put bar in /software/Core/bar but in something like /software2/Core/bar. (Using HierarchicalMNS naming scheme) You can always mix and match different methods, but if you want to *be aware* of each other then you will have to do more work. EasyBuild provides a consistent stack, if you start randomly adding additional software and don't maintain the consistency then you are likely to run into problems. It is possible to tell EB that certain software packages will be provided externally to EasyBuild (see http://easybuild.readthedocs.io/en/latest/Using_external_modules.html). I think you are discussing though 2 different things, where software is installed and what the modules tree look like, these are separate issues. You can install software in the EB installation path without issue, but you will still have to create a module file for it, otherwise it is invisible to EB (and actually EB will happily remove your installation if it is asked to install it itself). Since you probably don't want that then you can build your own software in a separate space, this is really doesn't matter since it's the module file that will be presented to the user, and that's where you will need to ensure you have consistency. Is my understanding of this correct? Does this only happen if rebuild/force flag is used? EB only sees something as available if it can find the module file that would be created with the command. If the software is available via a module file found in the right place with the right name (under the current naming scheme!) then it is considered installed, otherwise it is not and anything in it's path to get there will be bulldozed (since it would simply see the existing installation as a failed effort). If the module file is available, you would have to use `--force` to force a (re)installation. Does it only happen for the app trying to rebuild (and/or dependencies)? --- e.g. is it safe to have both foo and bar under /software/Core as long as I never issue an easybuild command that would try to rebuild bar (either directly or as a dependency)? E.g., is the foo/bar example somewhat OK, but I only get burned if I e.g. built foo/1.2.3 with eb and foo/1.2.4 manually? You would not get burned in the scenario you describe unless you install foo/1.2.4, don't inform EB about it (i.e., declare it as an external module as above anywhere it appears as a dep) and then build something that has foo/1.2.4 as a dep. This is a lot of manual work though if foo frequently appears as a dep. In my opinion you need to ask if you (verifiably) gain a lot by building it yourself. Is there a list anywhere of the directories that EasyBuild wants full control over? Is it the whole tree rooted at --prefix, or just the source, build, software install, module install paths? to Would it be safe to have foo in /software/[Core|Compiler| MPI]/foo and bar directly under /software? You can separate things as you like, see http://easybuild.readthedocs.io/en/latest/Configuration.html#mandatory-configuration-settings . The core issue is, who cares where things are installed, it's what's in the module files that matters. Better to be safe than sorry, separate out the EB installs so you don't to worry about these details then spend your time trying to make your modules consistent. Alan -- Dr. Alan O'Cais E-CAM Software Manager Juelich Supercomputing Centre Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49 2461 61 5213 Fax: +49 2461 61 6656 E-mail: a.oc...@fz-juelich.de<mailto:a.oc...@fz-juelich.de> WWW: http://www.fz-juelich.de/ias/jsc/EN ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
