Re: [easybuild] Lmod integration - Families

[Markus Geimer]

Davide,

> Do you add the family attribute manually after EB installed
> the default module for a specific package?

We currently add

modluafooter = 'family("compiler")'

or similar to the easyconfig file.

> For this purpose it would be nice to have a function in EB that would
> allow from the easyconfig file

There has been some work on this, see
https://github.com/hpcugent/easybuild-framework/pull/1257
However, the PR hasn't been touched for a while and will need some
care.  Maybe someone at the User Group meeting is keep on adopting
it ;-)

> (or even better in a easyconfig patch
> files)  to add additional lines to the generated module for a more easy
> to manage localization.

+1 for easyconfig patch files

Markus


> On Feb 6 2017, at 12:05 pm, Alvarez, Damian 
> wrote:
> 
> I’ll cover it a bit, but not too deeply. In any case, I don’t think
> Davide will come.
> 
>  
> 
> Anyway, what are the issues you are seeing? I think that using a
> toolchain based module naming scheme this would work similarly to
> the hierarchical scheme (ie: it will prevent you from loading a
> compiler on top of another one)
> 
>  
> 
> Damian
> 
>  
> 
> *From: * on behalf of Alan O'Cais
> 
> *Reply-To: *"easybuild@lists.ugent.be" 
> *Date: *Monday 6 February 2017 at 18:22
> *To: *"easybuild@lists.ugent.be" 
> *Subject: *Re: [easybuild] Lmod integration - Families
> 
>  
> 
> We use families for  compilers and MPI implementations in a
> hierarchical scheme. We do not expose the toolchains to normal
> users. Damian will probably cover this a little in his presentation
> on Wednesday.
> 
>  
> 
> Alan
> 
>  
> 
> On 6 February 2017 at 16:57, Vanzo, Davide
> >
> wrote:
> 
> Hello all,
> 
> one of the most useful features provided by Lmod is the ability
> to assign a family to a subset of modules in order to avoid
> loading potentially conflicting software. This gets a little
> complicated when dealing with the toolchain-based model of EB. I
> was wondering if anybody is using families and how.
> 
> Thank you.
> 
> 
> -- 
> Davide Vanzo, PhD
> Application Developer
> 
> Adjunct Assistant Professor of Chemical and Biomolecular Engineering
> 
> Advanced Computing Center for Research and Education (ACCRE)
> 
> Vanderbilt University - Hill Center 201
> www.accre.vanderbilt.edu 
> 
> 
> 
>  
> 
> -- 
> 
> Dr. Alan O'Cais
> E-CAM Software Manager
> Juelich Supercomputing Centre
> Forschungszentrum Juelich GmbH
> 52425 Juelich, Germany
> 
> Phone: +49 2461 61 5213
> Fax: +49 2461 61 6656
> E-mail: a.oc...@fz-juelich.de 
> WWW:http://www.fz-juelich.de/ias/jsc/EN
> 
> 
> 
> 
> 
> 
> 
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> 
> 
> 
> 
> 
> 

-- 
Dr. Markus Geimer
Juelich Supercomputing Centre
Institute for Advanced Simulation
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone:  +49-2461-61-1773
Fax:+49-2461-61-6656
E-Mail: m.gei...@fz-juelich.de
Web:http://www.fz-juelich.de/jsc

Re: [easybuild] Lmod integration - Families

[Vanzo, Davide]

Damian and Alex,
I am still on a pre-production stage and I am trying to find the best way to 
implement it in production such that the users experience will be the best 
possible. So I was just wandering what other groups do in order to avoid 
reinventing the wheel.
Your idea of not exposing the toolchain to users is already a good feedback. Do 
you add the family attribute manually after EB installed the default module for 
a specific package?
For this purpose it would be nice to have a function in EB that would allow 
from the easyconfig file (or even better in a easyconfig patch files)  to add 
additional lines to the generated module for a more easy to manage localization.

PS: You are right, I will not be able to join you guys at the user meeting. I 
will surely watch any recordings or slides you will make available.

Davide


On Feb 6 2017, at 12:05 pm, Alvarez, Damian  wrote:

I’ll cover it a bit, but not too deeply. In any case, I don’t think Davide will 
come.



Anyway, what are the issues you are seeing? I think that using a toolchain 
based module naming scheme this would work similarly to the hierarchical scheme 
(ie: it will prevent you from loading a compiler on top of another one)



Damian



From:  on behalf of Alan O'Cais 

Reply-To: "easybuild@lists.ugent.be" 
Date: Monday 6 February 2017 at 18:22
To: "easybuild@lists.ugent.be" 
Subject: Re: [easybuild] Lmod integration - Families



We use families for  compilers and MPI implementations in a hierarchical 
scheme. We do not expose the toolchains to normal users. Damian will probably 
cover this a little in his presentation on Wednesday.



Alan



On 6 February 2017 at 16:57, Vanzo, Davide 
> wrote:

Hello all,

one of the most useful features provided by Lmod is the ability to assign a 
family to a subset of modules in order to avoid loading potentially conflicting 
software. This gets a little complicated when dealing with the toolchain-based 
model of EB. I was wondering if anybody is using families and how.

Thank you.

--
Davide Vanzo, PhD
Application Developer

Adjunct Assistant Professor of Chemical and Biomolecular Engineering

Advanced Computing Center for Research and Education (ACCRE)

Vanderbilt University - Hill Center 201
www.accre.vanderbilt.edu





--

Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

Re: [easybuild] Lmod integration - Families

[Alvarez, Damian]

I’ll cover it a bit, but not too deeply. In any case, I don’t think Davide will 
come.

Anyway, what are the issues you are seeing? I think that using a toolchain 
based module naming scheme this would work similarly to the hierarchical scheme 
(ie: it will prevent you from loading a compiler on top of another one)

Damian

From:  on behalf of Alan O'Cais 

Reply-To: "easybuild@lists.ugent.be" 
Date: Monday 6 February 2017 at 18:22
To: "easybuild@lists.ugent.be" 
Subject: Re: [easybuild] Lmod integration - Families

We use families for  compilers and MPI implementations in a hierarchical 
scheme. We do not expose the toolchains to normal users. Damian will probably 
cover this a little in his presentation on Wednesday.

Alan

On 6 February 2017 at 16:57, Vanzo, Davide 
> wrote:
Hello all,
one of the most useful features provided by Lmod is the ability to assign a 
family to a subset of modules in order to avoid loading potentially conflicting 
software. This gets a little complicated when dealing with the toolchain-based 
model of EB. I was wondering if anybody is using families and how.
Thank you.

--
Davide Vanzo, PhD
Application Developer
Adjunct Assistant Professor of Chemical and Biomolecular Engineering
Advanced Computing Center for Research and Education (ACCRE)
Vanderbilt University - Hill Center 201
www.accre.vanderbilt.edu



--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

Re: [easybuild] Lmod integration - Families

[Alan O'Cais]

We use families for  compilers and MPI implementations in a hierarchical 
scheme. We do not expose the toolchains to normal users. Damian will probably 
cover this a little in his presentation on Wednesday.

Alan

On 6 February 2017 at 16:57, Vanzo, Davide 
> wrote:
Hello all,
one of the most useful features provided by Lmod is the ability to assign a 
family to a subset of modules in order to avoid loading potentially conflicting 
software. This gets a little complicated when dealing with the toolchain-based 
model of EB. I was wondering if anybody is using families and how.
Thank you.

--
Davide Vanzo, PhD
Application Developer
Adjunct Assistant Professor of Chemical and Biomolecular Engineering
Advanced Computing Center for Research and Education (ACCRE)
Vanderbilt University - Hill Center 201
www.accre.vanderbilt.edu



--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

[easybuild] Is there a way to discover which toolchains are currently installed and what modules they comprise of ?

[Maxime Boissonneault]

Hi all,

Is there a way for easybuild to tell me what are the toolchains which it 
currently sees as installed on the system ?


Note that we have a custom naming scheme, and that we hide toolchains to 
users.



Thanks,


--
-
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique

[easybuild] Lmod integration - Families

[Vanzo, Davide]

Hello all,
one of the most useful features provided by Lmod is the ability to assign a 
family to a subset of modules in order to avoid loading potentially conflicting 
software. This gets a little complicated when dealing with the toolchain-based 
model of EB. I was wondering if anybody is using families and how.
Thank you.

--
Davide Vanzo, PhD
Application Developer
Adjunct Assistant Professor of Chemical and Biomolecular Engineering
Advanced Computing Center for Research and Education (ACCRE)
Vanderbilt University - Hill Center 201
www.accre.vanderbilt.edu

Re: [easybuild] building mxnet

[Jure Pečar]

On Sat, 28 Jan 2017 17:43:42 +0100
Kenneth Hoste  wrote:

> If there are releases available for each of the submodules, you can make 
> EB pulls those in, and then you could reassemble things in the build dir 
> using symlinks via 'preconfigopts' for example in the easyconfig?
> 
> That would be one way to automate this while retaining control over 
> which sources are actually used...
 
Interesting idea. Will play with it when I get some time.
 
> You could just get that R library built via the extensions mechanism in 
> EasyBuild, and just make it part of the same installation?

I'm exploring this now ... and have a bit of a chicken-and-egg issue.

If you check mxnet docs at 
http://mxnet.io/get_started/ubuntu_setup.html#install-the-mxnet-package-for-r
you'll see that the steps to install the R-package of mxnet are:
* build mxnet
* Rscript install_deps (which installs about 11 additional R packages beside 
the ones listed in R eb)
* make rpkg (which installs mxnet R package and only then creates a tarball ... 
and tarball is created by R from mxnet:::mxnet.export ... grrr)
* at that point I have a tarball that I can push to sources dir and use as a 
source of R extension

I don't see how I could simply map this to existing easybuild facilities.

Maybe I can tear apart the "make rpkg" thing and try to reimplement it in .eb 
file ... But still I have to get those required R packages installed first. Any 
suggestions?

Maybe we can discuss this on Thursday ...


-- 

  Jurij Pečar
  HPC Engineer, IT Operations, IT Services
  EMBL Heidelberg, Meyerhofstraße 1, 69117, Heidelberg, Germany
  Room 13-401