Re: [easybuild] Find hidden modules by default
Hi Davide, On 04/08/16 15:31, Vanzo, Davide wrote: Thank you guys for your replies. I'm happy to see I'm not the only one that would benefit from this. I totally agree with Kenneth on how this should be implemented. Am I wrong to assume that a positive side effect of this approach is to make the "hiddendependencies" option for easyconfig files redundant since EB will automatically look for hidden dependencies if no visible ones are found for the ones listed in "dependencies" and "builddependencies"? Yes, that would probably make 'hiddendependencies' mostly redundant, but not entirely... Using 'hiddendependencies' means that those dependencies will always be hidden, regardless of the EasyBuild configuration being used. When we make the robot also consider hidden modules, this will be configurable (and probably enabled by default in EasyBuild v3.0 which should materialize later this year). So, depending on your use case, 'hiddendependencies' may still be useful, e.g. when you want to hide dependencies for something in particular, but not as a general approach. regards, Kenneth DV -- Davide Vanzo, PhD Application Developer Adjunct Assistant Professor of Chemical and Biomolecular Engineering Advanced Computing Center for Research and Education (ACCRE) Vanderbilt University - Hill Center 201 (615)-875-9137 www.accre.vanderbilt.edu On Aug 4 2016, at 5:11 am, Kenneth Hostewrote: Hi Markus, On 04/08/16 10:31, Markus Geimer wrote: > All, > > It seems that with [2] in place, the dependency resolution already > does "the right thing" as it parses the easyconfig files. But that > obviously only works if the easyconfig contains the 'hidden' > parameter. That said, maybe it is sufficient to ensure writing the > 'hidden' info (wherever it comes from: 'hidden' flag, 'hide-deps', > '--hidden' command-line option, ...) to the archived easyconfig? > Or am I missing something? That's probably only a partial solution, since you have no guarantee that the archived easyconfig will be picked up the next time the same installation is needed (the path to the archived easyconfigs is not in the robot path, by default). I *think* that making the robot consider also hidden modules is as simple as updating the 'find_resolved_modules' function to optionally check also whether the module is available as hidden (which involves tweaking the dep_mod_name to inject a '.', or something like that). You may need to make a similar change to make dry-run spit out the correct info though, which probably means some refactoring to ensure that both the robot and dry-run use the same function to check whether a module is available or not... regards, Kenneth > > Markus > > > [2] https://github.com/hpcugent/easybuild-framework/pull/1837 > > On 08/04/16 08:58, Markus Geimer wrote: >> Davide, >> >>> The most logical way of doing so seems to be by creating hidden >>> modules for the dependencies we don't want the users to see. However, >>> when installing other easyconfig packages from the default easyconfig >>> files they cannot see the hidden modules and try to install them again. >>> Is there a way to tell EB to automatically look for hidden packages >>> without modifying the easyconfig files? >> At this point not, unfortunately. A corresponding issue has >> been created already quite some time ago [1], but since then >> hasn't received much attention. Since I'm very interested in >> this feature as well and would like to see it implemented >> rather sooner than later, I recently started looking into it. >> But I'm still far from a clean and working solution -- though >> Kenneth claims that it shouldn't be too hard... But that's >> what he is saying all the time ;-) >> >> Markus >> >> >> [1] https://github.com/hpcugent/easybuild-framework/issues/1079 > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-mail: m.gei...@fz-juelich.de > WWW: http://www.fz-juelich.de/jsc/ > > > > > > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender),
Re: [easybuild] Find hidden modules by default
Thank you guys for your replies. I'm happy to see I'm not the only one that would benefit from this. I totally agree with Kenneth on how this should be implemented. Am I wrong to assume that a positive side effect of this approach is to make the "hiddendependencies" option for easyconfig files redundant since EB will automatically look for hidden dependencies if no visible ones are found for the ones listed in "dependencies" and "builddependencies"? DV -- Davide Vanzo, PhD Application Developer Adjunct Assistant Professor of Chemical and Biomolecular Engineering Advanced Computing Center for Research and Education (ACCRE) Vanderbilt University - Hill Center 201 (615)-875-9137 www.accre.vanderbilt.edu On Aug 4 2016, at 5:11 am, Kenneth Hostewrote: Hi Markus, On 04/08/16 10:31, Markus Geimer wrote: > All, > > It seems that with [2] in place, the dependency resolution already > does "the right thing" as it parses the easyconfig files. But that > obviously only works if the easyconfig contains the 'hidden' > parameter. That said, maybe it is sufficient to ensure writing the > 'hidden' info (wherever it comes from: 'hidden' flag, 'hide-deps', > '--hidden' command-line option, ...) to the archived easyconfig? > Or am I missing something? That's probably only a partial solution, since you have no guarantee that the archived easyconfig will be picked up the next time the same installation is needed (the path to the archived easyconfigs is not in the robot path, by default). I *think* that making the robot consider also hidden modules is as simple as updating the 'find_resolved_modules' function to optionally check also whether the module is available as hidden (which involves tweaking the dep_mod_name to inject a '.', or something like that). You may need to make a similar change to make dry-run spit out the correct info though, which probably means some refactoring to ensure that both the robot and dry-run use the same function to check whether a module is available or not... regards, Kenneth > > Markus > > > [2] https://github.com/hpcugent/easybuild-framework/pull/1837 > > On 08/04/16 08:58, Markus Geimer wrote: >> Davide, >> >>> The most logical way of doing so seems to be by creating hidden >>> modules for the dependencies we don't want the users to see. However, >>> when installing other easyconfig packages from the default easyconfig >>> files they cannot see the hidden modules and try to install them again. >>> Is there a way to tell EB to automatically look for hidden packages >>> without modifying the easyconfig files? >> At this point not, unfortunately. A corresponding issue has >> been created already quite some time ago [1], but since then >> hasn't received much attention. Since I'm very interested in >> this feature as well and would like to see it implemented >> rather sooner than later, I recently started looking into it. >> But I'm still far from a clean and working solution -- though >> Kenneth claims that it shouldn't be too hard... But that's >> what he is saying all the time ;-) >> >> Markus >> >> >> [1] https://github.com/hpcugent/easybuild-framework/issues/1079 > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-mail: m.gei...@fz-juelich.de > WWW: http://www.fz-juelich.de/jsc/ > > > > > > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > > >
Re: [easybuild] Find hidden modules by default
Hi Markus, On 04/08/16 10:31, Markus Geimer wrote: All, It seems that with [2] in place, the dependency resolution already does "the right thing" as it parses the easyconfig files. But that obviously only works if the easyconfig contains the 'hidden' parameter. That said, maybe it is sufficient to ensure writing the 'hidden' info (wherever it comes from: 'hidden' flag, 'hide-deps', '--hidden' command-line option, ...) to the archived easyconfig? Or am I missing something? That's probably only a partial solution, since you have no guarantee that the archived easyconfig will be picked up the next time the same installation is needed (the path to the archived easyconfigs is not in the robot path, by default). I *think* that making the robot consider also hidden modules is as simple as updating the 'find_resolved_modules' function to optionally check also whether the module is available as hidden (which involves tweaking the dep_mod_name to inject a '.', or something like that). You may need to make a similar change to make dry-run spit out the correct info though, which probably means some refactoring to ensure that both the robot and dry-run use the same function to check whether a module is available or not... regards, Kenneth Markus [2] https://github.com/hpcugent/easybuild-framework/pull/1837 On 08/04/16 08:58, Markus Geimer wrote: Davide, The most logical way of doing so seems to be by creating hidden modules for the dependencies we don't want the users to see. However, when installing other easyconfig packages from the default easyconfig files they cannot see the hidden modules and try to install them again. Is there a way to tell EB to automatically look for hidden packages without modifying the easyconfig files? At this point not, unfortunately. A corresponding issue has been created already quite some time ago [1], but since then hasn't received much attention. Since I'm very interested in this feature as well and would like to see it implemented rather sooner than later, I recently started looking into it. But I'm still far from a clean and working solution -- though Kenneth claims that it shouldn't be too hard... But that's what he is saying all the time ;-) Markus [1] https://github.com/hpcugent/easybuild-framework/issues/1079 -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax:+49-2461-61-6656 E-mail: m.gei...@fz-juelich.de WWW:http://www.fz-juelich.de/jsc/ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt
Re: [easybuild] Find hidden modules by default
The downside is that this is an "all or nothing" approach. For example, you may want to hide the M4/flex/Bison on the dummy level used for bootstrapping, but expose the same packages on the GCCcore level. No way to do this with 'hide-deps', AFAIK... Markus On 08/04/16 10:27, Alvarez, Damian wrote: > Well, alternatively you can add the dependencies to > EASYBUILD_HIDE_DEPS and have something like > EASYBUILD_HIDE_DEPS="ANTLR,ATK,Autoconf,Automake,Bison,etc”, and > those dependencies will be always hidden (so don’t install them > directly, or do it with --hidden). If the version number and the > suffix of your installed hidden dependencies match the dependencies > of the easyconfigs you want to install then it will work just fine. > > Damian > > On 04/08/16 08:58, "easybuild-requ...@lists.ugent.be on behalf of Markus > Geimer"m.gei...@fz-juelich.de> wrote: > > Davide, > > > The most logical way of doing so seems to be by creating hidden > > modules for the dependencies we don't want the users to see. However, > > when installing other easyconfig packages from the default easyconfig > > files they cannot see the hidden modules and try to install them again. > > Is there a way to tell EB to automatically look for hidden packages > > without modifying the easyconfig files? > > At this point not, unfortunately. A corresponding issue has > been created already quite some time ago [1], but since then > hasn't received much attention. Since I'm very interested in > this feature as well and would like to see it implemented > rather sooner than later, I recently started looking into it. > But I'm still far from a clean and working solution -- though > Kenneth claims that it shouldn't be too hard... But that's > what he is saying all the time ;-) > > Markus > > > [1] https://github.com/hpcugent/easybuild-framework/issues/1079 > > -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax:+49-2461-61-6656 E-mail: m.gei...@fz-juelich.de WWW:http://www.fz-juelich.de/jsc/ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt
Re: [easybuild] Find hidden modules by default
Well, alternatively you can add the dependencies to EASYBUILD_HIDE_DEPS and have something like EASYBUILD_HIDE_DEPS="ANTLR,ATK,Autoconf,Automake,Bison,etc”, and those dependencies will be always hidden (so don’t install them directly, or do it with --hidden). If the version number and the suffix of your installed hidden dependencies match the dependencies of the easyconfigs you want to install then it will work just fine. Damian On 04/08/16 08:58, "easybuild-requ...@lists.ugent.be on behalf of Markus Geimer"wrote: Davide, > The most logical way of doing so seems to be by creating hidden > modules for the dependencies we don't want the users to see. However, > when installing other easyconfig packages from the default easyconfig > files they cannot see the hidden modules and try to install them again. > Is there a way to tell EB to automatically look for hidden packages > without modifying the easyconfig files? At this point not, unfortunately. A corresponding issue has been created already quite some time ago [1], but since then hasn't received much attention. Since I'm very interested in this feature as well and would like to see it implemented rather sooner than later, I recently started looking into it. But I'm still far from a clean and working solution -- though Kenneth claims that it shouldn't be too hard... But that's what he is saying all the time ;-) Markus [1] https://github.com/hpcugent/easybuild-framework/issues/1079 -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax:+49-2461-61-6656 E-mail: m.gei...@fz-juelich.de WWW:http://www.fz-juelich.de/jsc/ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt
Re: [easybuild] Find hidden modules by default
Davide, > The most logical way of doing so seems to be by creating hidden > modules for the dependencies we don't want the users to see. However, > when installing other easyconfig packages from the default easyconfig > files they cannot see the hidden modules and try to install them again. > Is there a way to tell EB to automatically look for hidden packages > without modifying the easyconfig files? At this point not, unfortunately. A corresponding issue has been created already quite some time ago [1], but since then hasn't received much attention. Since I'm very interested in this feature as well and would like to see it implemented rather sooner than later, I recently started looking into it. But I'm still far from a clean and working solution -- though Kenneth claims that it shouldn't be too hard... But that's what he is saying all the time ;-) Markus [1] https://github.com/hpcugent/easybuild-framework/issues/1079 -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax:+49-2461-61-6656 E-mail: m.gei...@fz-juelich.de WWW:http://www.fz-juelich.de/jsc/ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt
[easybuild] Find hidden modules by default
Hello easybuilders, I am testing EB + Lmod with hierarchical naming and I am trying to reconcile the huge amount of modules generated by EB with the need of reducing users' confusion by showing only the strictly necessary modules. The most logical way of doing so seems to be by creating hidden modules for the dependencies we don't want the users to see. However, when installing other easyconfig packages from the default easyconfig files they cannot see the hidden modules and try to install them again. Is there a way to tell EB to automatically look for hidden packages without modifying the easyconfig files? Thank you -- Davide Vanzo, PhD Application Developer Adjunct Assistant Professor of Chemical and Biomolecular Engineering Advanced Computing Center for Research and Education (ACCRE) Vanderbilt University - Hill Center 201 (615)-875-9137 www.accre.vanderbilt.edu