date:20161013

Re: [easybuild] fpm error issues building GCC-5.2.0.eb

2016-10-13 Thread Kenneth Hoste


Hello,

On 13/10/16 17:36, shahzeb.siddi...@pfizer.com wrote:

I am getting an error during fpm process for GCC 5.2.0. I have eb set for
packaging by default.

I have given the output of the log where the error occurs.


Which EasyBuild & FPM versions are you using?

The only thing I can get from the error you're seeing is this:

Process failed: rpmbuild failed (exit code 1)

which is not very useful... Do you have rpmbuild available on your system?

Do you have any useful logs somewhere deeper in the temporary directory 
used by EasyBuild, i.e. /tmp/eb-lhMLNO in this case?

Maybe in /tmp/eb-lhMLNO/eb-pkgs-GIR_5q ?

I just tried to packaging GCC 5.2.0 myself using FPM 1.3.3 and an 
existing GCC 5.2.0 installation via "eb GCC-5.2.0.eb --package --skip 
--force", which worked fine (I was using the current EasyBuild develop 
version, but that shouldn't matter):


$ ls -l $EASYBUILD_PREFIX/packages
total 483648
-rw-rw-r-- 1 vsc40023 vsc40023 247605409 Oct 13 18:57 
GCC-5.2.0-eb_3.0.0.dev0-1.x86_64.rpm



regards,

Kenneth



[hpcswadm@amrndhl1155 ~]$ eb --show-full-config
#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment variable, F:
configuration file)
#
add-dummy-to-minimal-toolchains (D) = False
aggregate-regtest   (D) = None
allow-modules-tool-mismatch (D) = False
amend   (D) = None
avail-cfgfile-constants (D) = False
avail-easyconfig-constants  (D) = False
avail-easyconfig-licenses   (D) = False
avail-easyconfig-params (D) = False
avail-easyconfig-templates  (D) = False
avail-module-naming-schemes (D) = False
avail-modules-tools (D) = False
avail-repositories  (D) = False
avail-toolchain-opts(D) = None
buildpath   (F) = /nfs/grid/software/RHEL7-BUILD/
easybuild/build
check-conflicts (D) = False
check-github(D) = False
cleanup-builddir(D) = True
cleanup-tmpdir  (D) = True
color   (D) = auto
configfiles (D) = /hpc/hpcswadm/.config/easybuild/
config.cfg
debug   (D) = False
debug-lmod  (D) = False
default-opt-level   (D) = defaultopt
dep-graph   (D) = None
deprecated  (D) = None
download-timeout(D) = None
dry-run (D) = False
dry-run-short   (D) = False
dump-autopep8   (D) = False
dump-env-script (D) = False
dump-test-report(D) = None
easyblock   (D) = None
experimental(D) = False
extended-dry-run(D) = False
extended-dry-run-ignore-errors  (D) = True
external-modules-metadata   (D) = None
extra-modules   (D) = None
filter-deps (D) = None
fixed-installdir-naming-scheme  (D) = False
force   (D) = False
from-pr (D) = None
git-working-dirs-path   (D) = None
github-org  (D) = None
github-user (D) = None
group   (D) = None
group-writable-installdir   (D) = False
hidden  (D) = False
hide-deps   (D) = None
hide-toolchains (D) = None
ignore-dirs (D) = .git, .svn
ignore-osdeps   (D) = False
ignoreconfigfiles   (D) = None
include-easyblocks  (D) =
include-module-naming-schemes   (D) =
include-toolchains  (D) =
info(D) = False
install-github-token(D) = False
install-latest-eb-release   (D) = False
installpath (F) = /nfs/grid/software/RHEL7-apps/
installpath-modules (D) = None
installpath-software(D) = None
job (D) = False
job-backend (D) = PbsPython
job-backend-config  (D) = None
job-cores   (D) = None
job-max-walltime(D) = 24
job-output-dir  (D) = /hpc/hpcswadm
job-polling-interval(D) = 30.0
job-target-resource (D) = None
last-log(D) = False
list-easyblocks (D) = None
list-installed-software (D) = None
list-software   (D) = None
list-toolchains (D) = False
logfile-format  (D) = ('easybuild', 'easybuild-%(name)s-%
(version)s-%(date)s.%(time)s.log')
logtostdout (D) = False
minimal-toolchains  (D) = False
module-naming-scheme(D) = EasyBuildMNS
module-only (D) = False
module-syntax   (D) = Tcl
moduleclasses   (D) = base, bio, cae, chem, compiler, data,
debugger, devel, geo, ide, lang, lib

Re: [easybuild] no module for CUDA-7.5.18

2016-10-13 Thread Kenneth Hoste

Hi Damian,

On 13/10/16 18:04, Alvarez, Damian wrote:

Well, you can simply put CUDA as a dependency in the OpenMPI easyconfig, right? 
You don’t really need to have a gcccuda toolchain for it, GCC would work just 
fine.

Sure, the idea of gcccuda is that EasyBuild can use that as a toolchain; 
it knows how to use the CUDA compiler, but this never really took off.

Also, as far as I remember, the CUDA installation doesn’t actively check which 
version of GCC you are using. That is done at compile time via 
include/host_config.h. Or am I missing something?

OK, I may be mistaken here, I was recalling from memory.

K.

Damian

On 13/10/16 16:53, "easybuild-requ...@lists.ugent.be on behalf of Kenneth Hoste" 
 wrote:

 On 11/10/16 15:54, Joachim Hein wrote:
 > Hi Åke and everyone else:
 >
 > Many thanks that builds.  (Only tried the cuda easy config, not the rest 
yet).As far as I can see the key difference is, that you package uses a 
GCC-5.4.0-2.26 toolchain while the standard Cuda 7.5.18 that comes with EB 2.9.0 
utilises a dummy toolchain.   So I am wondering whether the Cuda modules in EB 
should be moved to a non-dummy toolchain.
 >
 > Our systems gcc is 4.8.5 (CentOS 7).  Though there might be an issue 
with the GPU node (missing package) I am using to build Cuda.  Normally I use a 
different node for EB work.  Any comments/suggestions?

 Well, there's a difference between an OpenMPI built on top of CUDA, or
 just installing them side-by-side.

 See for example

https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.7.3-gcccuda-2.6.10.eb
 that includes this:

  configopts += '--with-cuda=$CUDA_HOME ' # CUDA-aware
 build; N.B. --disable-dlopen is incompatible

 The OpenMPI easyconfig in
 https://github.com/hpcugent/easybuild-easyconfigs/pull/3666 also
 includes this.

 Whether you're installing CUDA on top of a GCC or not doesn't really
 matter though, since it's a binary installation.

 The only advantage (I can think of) is that the CUDA installer checks
 the GCC version being used.
 If you install CUDA with a dummy toolchain, it'll check the version of
 the system GCC, after which you'll sneakily combine that CUDA
 installation with a different GCC provided via EasyBuild.

 regards,

 Kenneth
 >
 > Best wishes
 >Joachim
 >
 >
 >
 >> On 11 Oct 2016, at 14:19, Åke Sandgren  
wrote:
 >>
 >> It's easyconfig PR 3666
 >>
 >> On 10/11/2016 12:34 PM, Joachim Hein wrote:
 >>> Hi Åke,
 >>>
 >>> Thanks for your reply.  For that you must have created a cuda 8.0.44 
configuration file.  Could you share that in some way (or is it in github in some place)?
 >>>
 >>> Best wishes
 >>>   Joachim
 >>>
 >>>
  On 11 Oct 2016, at 12:21, Åke Sandgren  
wrote:

  No clue on the 7.5.18, but i've just made a goolfc toolchain based on
  cuda 8.0.44, gcc 5.4.0 (they still don't support any newer version of
  gcc) and the latest fftw/scalapack/openblas

  That one seemed to build at least, haven't had time to test it.

  On 10/11/2016 11:57 AM, Joachim Hein wrote:
 > Hi,
 >
 > We are still struggling to build a CUDA 7.5.18 with EB 2.9.0.  It 
seems
 > to be failing in the “sanity check”, though the actual nvidia 
installer
 > doesn’t give an error and the bin directory looks reasonable to us.  
We
 > get a “KeyError: ‘GCC’
 >
 > Anyone could comment with regards to what went wrong:
 >
 > -bash-4.2$ eb CUDA-7.5.18.eb
 > == temporary log file in case of crash 
/tmp/eb-wjA2JG/easybuild-2WrNil.log
 > == processing EasyBuild easyconfig
 > 
/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.9.0-py2.7.egg/easybuild/easyconfigs/c/CUDA/CUDA-7.5.18.eb
 > == building and installing Core/CUDA/7.5.18...
 > == fetching files...
 > == creating build dir, resetting environment...
 > == unpacking...
 > == patching...
 > == preparing...
 > == configuring...
 > == building...
 > == testing...
 > == installing...
 > == taking care of extensions...
 > == postprocessing...
 > == sanity checking...
 > ERROR: Traceback (most recent call last):
 > File
 > 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/main.py",
 > line 115, in build_and_install_software
 >(ec_res['success'], app_log, err) = build_and_install_one(ec, 
init_env)
 > File
 > 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/ea

Re: [easybuild] no module for CUDA-7.5.18

2016-10-13 Thread Alvarez, Damian

Well, you can simply put CUDA as a dependency in the OpenMPI easyconfig, right? 
You don’t really need to have a gcccuda toolchain for it, GCC would work just 
fine.

Also, as far as I remember, the CUDA installation doesn’t actively check which 
version of GCC you are using. That is done at compile time via 
include/host_config.h. Or am I missing something?

Damian

On 13/10/16 16:53, "easybuild-requ...@lists.ugent.be on behalf of Kenneth 
Hoste"  
wrote:

On 11/10/16 15:54, Joachim Hein wrote:
> Hi Åke and everyone else:
>
> Many thanks that builds.  (Only tried the cuda easy config, not the rest 
yet).As far as I can see the key difference is, that you package uses a 
GCC-5.4.0-2.26 toolchain while the standard Cuda 7.5.18 that comes with EB 
2.9.0 utilises a dummy toolchain.   So I am wondering whether the Cuda modules 
in EB should be moved to a non-dummy toolchain.
>
> Our systems gcc is 4.8.5 (CentOS 7).  Though there might be an issue with 
the GPU node (missing package) I am using to build Cuda.  Normally I use a 
different node for EB work.  Any comments/suggestions?

Well, there's a difference between an OpenMPI built on top of CUDA, or
just installing them side-by-side.

See for example

https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.7.3-gcccuda-2.6.10.eb
that includes this:

 configopts += '--with-cuda=$CUDA_HOME ' # CUDA-aware
build; N.B. --disable-dlopen is incompatible

The OpenMPI easyconfig in
https://github.com/hpcugent/easybuild-easyconfigs/pull/3666 also
includes this.


Whether you're installing CUDA on top of a GCC or not doesn't really
matter though, since it's a binary installation.

The only advantage (I can think of) is that the CUDA installer checks
the GCC version being used.
If you install CUDA with a dummy toolchain, it'll check the version of
the system GCC, after which you'll sneakily combine that CUDA
installation with a different GCC provided via EasyBuild.


regards,

Kenneth
>
> Best wishes
>Joachim
>
>
>
>> On 11 Oct 2016, at 14:19, Åke Sandgren  wrote:
>>
>> It's easyconfig PR 3666
>>
>> On 10/11/2016 12:34 PM, Joachim Hein wrote:
>>> Hi Åke,
>>>
>>> Thanks for your reply.  For that you must have created a cuda 8.0.44 
configuration file.  Could you share that in some way (or is it in github in 
some place)?
>>>
>>> Best wishes
>>>   Joachim
>>>
>>>
 On 11 Oct 2016, at 12:21, Åke Sandgren  
wrote:

 No clue on the 7.5.18, but i've just made a goolfc toolchain based on
 cuda 8.0.44, gcc 5.4.0 (they still don't support any newer version of
 gcc) and the latest fftw/scalapack/openblas

 That one seemed to build at least, haven't had time to test it.

 On 10/11/2016 11:57 AM, Joachim Hein wrote:
> Hi,
>
> We are still struggling to build a CUDA 7.5.18 with EB 2.9.0.  It 
seems
> to be failing in the “sanity check”, though the actual nvidia 
installer
> doesn’t give an error and the bin directory looks reasonable to us.  
We
> get a “KeyError: ‘GCC’
>
> Anyone could comment with regards to what went wrong:
>
> -bash-4.2$ eb CUDA-7.5.18.eb
> == temporary log file in case of crash 
/tmp/eb-wjA2JG/easybuild-2WrNil.log
> == processing EasyBuild easyconfig
> 
/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.9.0-py2.7.egg/easybuild/easyconfigs/c/CUDA/CUDA-7.5.18.eb
> == building and installing Core/CUDA/7.5.18...
> == fetching files...
> == creating build dir, resetting environment...
> == unpacking...
> == patching...
> == preparing...
> == configuring...
> == building...
> == testing...
> == installing...
> == taking care of extensions...
> == postprocessing...
> == sanity checking...
> ERROR: Traceback (most recent call last):
> File
> 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/main.py",
> line 115, in build_and_install_software
>(ec_res['success'], app_log, err) = build_and_install_one(ec, 
init_env)
> File
> 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py",
> line 2376, in build_and_install_one
>result = app.run_all_steps(run_test_cases=run_test_cases)
> File
> 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py",
> line 2292, in run_all_ste

[easybuild] fpm error issues building GCC-5.2.0.eb

2016-10-13 Thread shahzeb . siddiqui

I am getting an error during fpm process for GCC 5.2.0. I have eb set for
packaging by default.

I have given the output of the log where the error occurs.


[hpcswadm@amrndhl1155 ~]$ eb --show-full-config
#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment variable, F:
configuration file)
#
add-dummy-to-minimal-toolchains (D) = False
aggregate-regtest   (D) = None
allow-modules-tool-mismatch (D) = False
amend   (D) = None
avail-cfgfile-constants (D) = False
avail-easyconfig-constants  (D) = False
avail-easyconfig-licenses   (D) = False
avail-easyconfig-params (D) = False
avail-easyconfig-templates  (D) = False
avail-module-naming-schemes (D) = False
avail-modules-tools (D) = False
avail-repositories  (D) = False
avail-toolchain-opts(D) = None
buildpath   (F) = /nfs/grid/software/RHEL7-BUILD/
easybuild/build
check-conflicts (D) = False
check-github(D) = False
cleanup-builddir(D) = True
cleanup-tmpdir  (D) = True
color   (D) = auto
configfiles (D) = /hpc/hpcswadm/.config/easybuild/
config.cfg
debug   (D) = False
debug-lmod  (D) = False
default-opt-level   (D) = defaultopt
dep-graph   (D) = None
deprecated  (D) = None
download-timeout(D) = None
dry-run (D) = False
dry-run-short   (D) = False
dump-autopep8   (D) = False
dump-env-script (D) = False
dump-test-report(D) = None
easyblock   (D) = None
experimental(D) = False
extended-dry-run(D) = False
extended-dry-run-ignore-errors  (D) = True
external-modules-metadata   (D) = None
extra-modules   (D) = None
filter-deps (D) = None
fixed-installdir-naming-scheme  (D) = False
force   (D) = False
from-pr (D) = None
git-working-dirs-path   (D) = None
github-org  (D) = None
github-user (D) = None
group   (D) = None
group-writable-installdir   (D) = False
hidden  (D) = False
hide-deps   (D) = None
hide-toolchains (D) = None
ignore-dirs (D) = .git, .svn
ignore-osdeps   (D) = False
ignoreconfigfiles   (D) = None
include-easyblocks  (D) =
include-module-naming-schemes   (D) =
include-toolchains  (D) =
info(D) = False
install-github-token(D) = False
install-latest-eb-release   (D) = False
installpath (F) = /nfs/grid/software/RHEL7-apps/
installpath-modules (D) = None
installpath-software(D) = None
job (D) = False
job-backend (D) = PbsPython
job-backend-config  (D) = None
job-cores   (D) = None
job-max-walltime(D) = 24
job-output-dir  (D) = /hpc/hpcswadm
job-polling-interval(D) = 30.0
job-target-resource (D) = None
last-log(D) = False
list-easyblocks (D) = None
list-installed-software (D) = None
list-software   (D) = None
list-toolchains (D) = False
logfile-format  (D) = ('easybuild', 'easybuild-%(name)s-%
(version)s-%(date)s.%(time)s.log')
logtostdout (D) = False
minimal-toolchains  (D) = False
module-naming-scheme(D) = EasyBuildMNS
module-only (D) = False
module-syntax   (D) = Tcl
moduleclasses   (D) = base, bio, cae, chem, compiler, data,
debugger, devel, geo, ide, lang, lib, math, mpi, numlib, perf, phys, system,
toolchain, tools, vis
modules-footer  (D) = None
modules-header  (D) = None
modules-tool(D) = EnvironmentModulesC
mpi-cmd-template(D) = None
mpi-tests   (D) = True
new-pr  (D) = False
only-blocks (D) = None
optarch (D) = None
output-format   (D) = txt
package (F) = True
package-naming-scheme   (D) = EasyBuildPNS
package-release (D) = 1
package-tool(D) = fpm
package-type(D) = rpm
packagepath (F) = /nfs/grid/software/RHEL7-BUILD/
easybuild/packages
parallel(D) = None
pr-branch-name  (D) = None
pr-commit-msg

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Ole Holm Nielsen


On 10/13/2016 05:15 PM, Kenneth Hoste wrote:

You need to move the 'module --initial_load restore' after the 'module
use', otherwise the EasyBuild module is indeed not available.
No Catch-22 imho...

And then you shouldn't need to load the EasyBuild module again, it
should be loaded via the restore...


Confirmed!  This initialization script works correctly:

if [ -z "$__Init_Default_Modules" ]; then
   export __Init_Default_Modules=1
   export EASYBUILD_MODULES_TOOL=Lmod
   export EASYBUILD_PREFIX=/home/modules
   module use $EASYBUILD_PREFIX/modules/all
   export LMOD_SYSTEM_DEFAULT_MODULES=EasyBuild
   module --initial_load restore
...

Thanks,
Ole

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Kenneth Hoste




On 13/10/16 17:12, Ole Holm Nielsen wrote:

On 10/13/2016 05:01 PM, Kenneth Hoste wrote:

The recommended line "module --initial_load restore" generates an
unwarranted message:


The system default contains no modules
  (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty)
  No changes in loaded modules


so I got rid of it by commenting this line out.


Well, you need to define $LMOD_SYSTEM_DEFAULT_MODULES :)


Do I really *need* to?  What's the point, and what should I do then?


If you define $LMOD_SYSTEM_DEFAULT_MODULES=EasyBuild, then you (or your
users) can easily restore after purge using 'module restore'.


I'm probably doing something wrong, but if I define 
LMOD_SYSTEM_DEFAULT_MODULES before setting up EB,


if [ -z "$__Init_Default_Modules" ]; then
   export __Init_Default_Modules=1
   export LMOD_SYSTEM_DEFAULT_MODULES=EasyBuild
   module --initial_load restore
   export EASYBUILD_MODULES_TOOL=Lmod
   export EASYBUILD_PREFIX=/home/modules
   module use $EASYBUILD_PREFIX/modules/all
   module load EasyBuild
...

then I get an error:

Lmod has detected the following error:  The following module(s) are 
unknown: "EasyBuild"


Please check the spelling or version number. Also try "module spider ..."

So is there a Catch-22 situation here?


You need to move the 'module --initial_load restore' after the 'module 
use', otherwise the EasyBuild module is indeed not available.

No Catch-22 imho...

And then you shouldn't need to load the EasyBuild module again, it 
should be loaded via the restore...





One thing that comes to mind: do all your users have access to
/home/modules ($EASYBUILD_PREFIX)?


Yes, I've NFS-mounted /home/modules (a CPU-architecture dependent tree 
for different Xeon versions) globally, and the directory is 
world-readable.


/Ole

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Ole Holm Nielsen


On 10/13/2016 05:01 PM, Kenneth Hoste wrote:

The recommended line "module --initial_load restore" generates an
unwarranted message:


The system default contains no modules
  (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty)
  No changes in loaded modules


so I got rid of it by commenting this line out.


Well, you need to define $LMOD_SYSTEM_DEFAULT_MODULES :)


Do I really *need* to?  What's the point, and what should I do then?


If you define $LMOD_SYSTEM_DEFAULT_MODULES=EasyBuild, then you (or your
users) can easily restore after purge using 'module restore'.


I'm probably doing something wrong, but if I define 
LMOD_SYSTEM_DEFAULT_MODULES before setting up EB,


if [ -z "$__Init_Default_Modules" ]; then
   export __Init_Default_Modules=1
   export LMOD_SYSTEM_DEFAULT_MODULES=EasyBuild
   module --initial_load restore
   export EASYBUILD_MODULES_TOOL=Lmod
   export EASYBUILD_PREFIX=/home/modules
   module use $EASYBUILD_PREFIX/modules/all
   module load EasyBuild
...

then I get an error:

Lmod has detected the following error:  The following module(s) are 
unknown: "EasyBuild"


Please check the spelling or version number. Also try "module spider ..."

So is there a Catch-22 situation here?


One thing that comes to mind: do all your users have access to
/home/modules ($EASYBUILD_PREFIX)?


Yes, I've NFS-mounted /home/modules (a CPU-architecture dependent tree 
for different Xeon versions) globally, and the directory is world-readable.


/Ole

Re: [easybuild] matplotlib installation fails with ImportError

2016-10-13 Thread Maik Schmidt

Hi,

I noticed that in the site-packages directory of the Python
installation, there was only "numpy" but no corresponding *egg-info
file. Apparently, this caused setuptools to fetch the latest version
again during the matplotlib setup. I have no idea why the file was gone,
but after restoring it from one of our older file system snapshots, the
matplotlib installation now works fine.

Mystery solved, sorry for the confusion.

Kind regards,
Maik

Am 13.10.2016 um 12:25 schrieb Kenneth Hoste:

Hi Maik,

I just tried to reinstall matplotlib-1.5.1-intel-2016a-Python-3.5.1.eb
, and I'm not seeing that a newer numpy is being picked up at all...

Can you share the full log file with us, preferably collected using eb
--debug?

Is there any chance you have numpy 1.11.2 installed system-wide, or
somehow avaiable through $PYTHONPATH or in
$HOME/.local/lib/python*/site-packages?

regards,

Kenneth

On 13/10/16 12:13, Maik Schmidt wrote:

Kenneth,
I'm using the unmodified easyconfig files from the EasyBuild
distribution (just with a different toolchain as I mentioned in my
original mail).

https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/p/Python/Python-3.5.1-intel-2016a.eb

and
https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/m/matplotlib/matplotlib-1.5.1-intel-2016a-Python-3.5.1.eb

Ole,
yes, the Python 2 installation works for me, too. It's just Python 3
that seems to have this problem, no idea why...

Regards,
Maik

Am 13.10.2016 um 11:44 schrieb Ole Holm Nielsen:
FWIW, we build matplotlib without problems using the foss2016b
toolchain, see the attached easyconfig file, which is based upon the
distribution file matplotlib-1.5.2-intel-2016b-Python-2.7.12.eb.

/Ole

On 10/13/2016 11:39 AM, Kenneth Hoste wrote:

Hi Maik,

On 13/10/16 11:37, Maik Schmidt wrote:

Hi Kenneth,

thanks for your input. I have tried with matplotlib 1.5.2 and 1.5.3,
but unfortunately, the error remains the same.

The numpy version in my Python 3.5.1 is the original one from the
respective easyconfig, namely 1.10.4. The matplotlib installer even
says so under

REQUIRED DEPENDENCIES AND EXTENSIONS
numpy: yes [version 1.10.4]

however, during the installation process, it somehow seems to have
fetched and referenced a newer version of numpy (1.11.2) in a temp
directory as can be seen in the log:

/scratch/hpcsupport/easybuild-tmp/eb-i1onxx/easy_install-v0f9q0bi/numpy-1.11.2/setup.py

So I guess this is more a problem of the installer, always trying to
use the latest numpy version or so?

That's surprising... Why would it try to download/install it's own
numpy

when it found one? I've never seen that happening (but maybe I just
overlooked it).

Can you share the easyconfig files you're using?

regards,

Kenneth

Kind regards,
Maik

Am 13.10.2016 um 11:00 schrieb Kenneth Hoste:

Hi Maik,

On 13/10/16 10:43, Maik Schmidt wrote:

Hi all,

I've installed Python-3.5.1-intel-2016a.eb (with
--try-toolchain="intel,2016.03-GCC-5.3" but that shouldn't be
relevant to the problem) and am now trying to install
matplotlib-1.5.1-intel-2016a-Python-3.5.1.eb, but it always
fails on

installing the extension matplotlib.

== 2016-10-13 10:31:49,338 build_log.py:163 ERROR cmd "
/sw/taurus/eb/Python/3.5.1-intel-2016.03-GCC-5.3/bin/python
setup.py

build " exited with exitcode 1 and output: ...

From the log I can see that an import exception is raised:

ImportError: No module named 'numpy._build_utils'

I have attached the full log.

Is this normal? Why does it try to access numpy._build_utils
anways?

Isn't this just some internal module of the numpy installer and
shouldn't even be used by other modules? Any advice on how to get
rid of this problem would be much appreciated. Thanks in advance.

This looks relevant: https://github.com/numpy/numpy/issues/6551 .

It seems to suggest that the numpy version you have included in your
Python 3.5.1 installation is too new for matplotlib 1.5.1?

Maybe try with matplotlib 1.5.2, if that's an option?

regards,

Kenneth

--
Maik Schmidt

Technische Universität Dresden
Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH)
Willers-Bau A116
D-01062 Dresden
Telefon: +49 351 463-32836

smime.p7s
Description: S/MIME Cryptographic Signature

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Kenneth Hoste




On 13/10/16 16:42, Ole Holm Nielsen wrote:

On 10/13/2016 04:27 PM, Kenneth Hoste wrote:
I'm afraid I don't have one. We don't load Easybuild by default for 
our

users. But you should take a look at:
https://lmod.readthedocs.io/en/latest/070_standard_modules.html


Thanks, this page is actually useful and specific!  So I've created a
new file /etc/profile.d/z01_EasyBuild.sh (to be loaded *after* Lmod's
z00_* scripts) with this content:

if [ -z "$__Init_Default_Modules" ]; then
   export __Init_Default_Modules=1;
   # module --initial_load restore
   export EASYBUILD_MODULES_TOOL=Lmod
   export EASYBUILD_PREFIX=/home/modules
   module use $EASYBUILD_PREFIX/modules/all

Is this right?

Are the modules really in /home/modules/modules/all?


Yes, I NFS-mount /home/modules, and there are 5 EB directories there:

# ls -la /home/modules
total 16
drwxrwxr-x.  7 modules modules   80 Oct 13 12:17 .
drwxr-xr-x.  3 rootroot   0 Oct 13 16:34 ..
drwxrwxr-x.  2 modules modules6 Oct 13 14:21 build
drwxrwxr-x. 35 modules modules 4096 Oct 13 14:21 ebfiles_repo
drwxrwxr-x. 15 modules modules 4096 Oct 13 13:55 modules
drwxrwxr-x. 36 modules modules 4096 Oct 13 14:21 software
drwxrwxr-x. 16 modules modules 4096 Sep 29 15:28 sources

Are you suggesting that only the "modules" subdirectory will actually 
be needed by users?  Anyway, the EB module building requires access to 
all subdirectories, so it seems convenient to have all subdirectories 
exposed this way.  Am I making any mistakes here?


No, it just looks weird to me to have a modules/modules directory, seems 
like a potential source for confusion...


I'd personally go with /home/apps or /apps or /home/easybuild, but 
that's up to you :)





The recommended line "module --initial_load restore" generates an
unwarranted message:


The system default contains no modules
  (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty)
  No changes in loaded modules


so I got rid of it by commenting this line out.


Well, you need to define $LMOD_SYSTEM_DEFAULT_MODULES :)


Do I really *need* to?  What's the point, and what should I do then?


If you define $LMOD_SYSTEM_DEFAULT_MODULES=EasyBuild, then you (or your 
users) can easily restore after purge using 'module restore'.


Other than that, I don't think there are big advantages, but someone 
will correct me if I'm wrong (e.g. Robert in CC, the Lmod lead developer).





Question: Does anyone see potential problems or bad side effects with
such a global setup of EB for all users?


Having a proper default setup in place makes sense, but users can easily
override it with their own config file or via environment variables.


That sounds like a "Good Thing" for advanced users, right?


Sure. If the users are indeed advanced, I guess they can potentially 
shoot themselves in the foot easily too by not realizing what redefining 
$EASYBUILD_PREFIX or undefining $EASYBUILD_MODULES_TOOL means... They'll 
notice soon enough I guess. ;)





The only thing we do is load Lmod itself (a symlink to init/bash in
/etc/profile.d).


Question: How do your users initialize the usage of EB?


They can define a config file, or define environment variables
(typically via .bashrc), up to them...


Ah, so it's DIY for your users!  I'm trying to make it easy for new 
users so that they don't need to manipulate .bashrc after reading some 
web documentation...  I guess that's a matter of taste.
Well, only users that are, for whatever reason, not happy with the 
system-level configuration you put in place would do that, of course.


One thing that comes to mind: do all your users have access to 
/home/modules ($EASYBUILD_PREFIX)?



regards,

Kenneth

Re: [easybuild] no module for CUDA-7.5.18

2016-10-13 Thread Kenneth Hoste


On 11/10/16 15:54, Joachim Hein wrote:

Hi Åke and everyone else:

Many thanks that builds.  (Only tried the cuda easy config, not the rest yet).  
  As far as I can see the key difference is, that you package uses a 
GCC-5.4.0-2.26 toolchain while the standard Cuda 7.5.18 that comes with EB 
2.9.0 utilises a dummy toolchain.   So I am wondering whether the Cuda modules 
in EB should be moved to a non-dummy toolchain.

Our systems gcc is 4.8.5 (CentOS 7).  Though there might be an issue with the 
GPU node (missing package) I am using to build Cuda.  Normally I use a 
different node for EB work.  Any comments/suggestions?


Well, there's a difference between an OpenMPI built on top of CUDA, or 
just installing them side-by-side.


See for example 
https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.7.3-gcccuda-2.6.10.eb 
that includes this:


configopts += '--with-cuda=$CUDA_HOME ' # CUDA-aware 
build; N.B. --disable-dlopen is incompatible


The OpenMPI easyconfig in 
https://github.com/hpcugent/easybuild-easyconfigs/pull/3666 also 
includes this.



Whether you're installing CUDA on top of a GCC or not doesn't really 
matter though, since it's a binary installation.


The only advantage (I can think of) is that the CUDA installer checks 
the GCC version being used.
If you install CUDA with a dummy toolchain, it'll check the version of 
the system GCC, after which you'll sneakily combine that CUDA 
installation with a different GCC provided via EasyBuild.



regards,

Kenneth


Best wishes
   Joachim




On 11 Oct 2016, at 14:19, Åke Sandgren  wrote:

It's easyconfig PR 3666

On 10/11/2016 12:34 PM, Joachim Hein wrote:

Hi Åke,

Thanks for your reply.  For that you must have created a cuda 8.0.44 
configuration file.  Could you share that in some way (or is it in github in 
some place)?

Best wishes
  Joachim



On 11 Oct 2016, at 12:21, Åke Sandgren  wrote:

No clue on the 7.5.18, but i've just made a goolfc toolchain based on
cuda 8.0.44, gcc 5.4.0 (they still don't support any newer version of
gcc) and the latest fftw/scalapack/openblas

That one seemed to build at least, haven't had time to test it.

On 10/11/2016 11:57 AM, Joachim Hein wrote:

Hi,

We are still struggling to build a CUDA 7.5.18 with EB 2.9.0.  It seems
to be failing in the “sanity check”, though the actual nvidia installer
doesn’t give an error and the bin directory looks reasonable to us.  We
get a “KeyError: ‘GCC’

Anyone could comment with regards to what went wrong:

-bash-4.2$ eb CUDA-7.5.18.eb
== temporary log file in case of crash /tmp/eb-wjA2JG/easybuild-2WrNil.log
== processing EasyBuild easyconfig
/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.9.0-py2.7.egg/easybuild/easyconfigs/c/CUDA/CUDA-7.5.18.eb
== building and installing Core/CUDA/7.5.18...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
ERROR: Traceback (most recent call last):
File
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/main.py",
line 115, in build_and_install_software
   (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
File
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py",
line 2376, in build_and_install_one
   result = app.run_all_steps(run_test_cases=run_test_cases)
File
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py",
line 2292, in run_all_steps
   self.run_step(step_name, step_methods)
File
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py",
line 2171, in run_step
   step_method(self)()
File
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_easyblocks-2.9.0-py2.7.egg/easybuild/easyblocks/c/cuda.py",
line 137, in sanity_check_step
   super(EB_CUDA, self).sanity_check_step(custom_paths=custom_paths)
File
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py",
line 1790, in sanity_check_step
   self._sanity_check_step(*args, **kwargs)
File
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py",
line 1910, in _sanity_check_step
   fake_mod_data = self.load_fake_module(purge=True)
File
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py",
line 1183, in load_fake_module
   fake_mo

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Ole Holm Nielsen


On 10/13/2016 04:27 PM, Kenneth Hoste wrote:

I'm afraid I don't have one. We don't load Easybuild by default for our
users. But you should take a look at:
https://lmod.readthedocs.io/en/latest/070_standard_modules.html


Thanks, this page is actually useful and specific!  So I've created a
new file /etc/profile.d/z01_EasyBuild.sh (to be loaded *after* Lmod's
z00_* scripts) with this content:

if [ -z "$__Init_Default_Modules" ]; then
   export __Init_Default_Modules=1;
   # module --initial_load restore
   export EASYBUILD_MODULES_TOOL=Lmod
   export EASYBUILD_PREFIX=/home/modules
   module use $EASYBUILD_PREFIX/modules/all

Is this right?

Are the modules really in /home/modules/modules/all?


Yes, I NFS-mount /home/modules, and there are 5 EB directories there:

# ls -la /home/modules
total 16
drwxrwxr-x.  7 modules modules   80 Oct 13 12:17 .
drwxr-xr-x.  3 rootroot   0 Oct 13 16:34 ..
drwxrwxr-x.  2 modules modules6 Oct 13 14:21 build
drwxrwxr-x. 35 modules modules 4096 Oct 13 14:21 ebfiles_repo
drwxrwxr-x. 15 modules modules 4096 Oct 13 13:55 modules
drwxrwxr-x. 36 modules modules 4096 Oct 13 14:21 software
drwxrwxr-x. 16 modules modules 4096 Sep 29 15:28 sources

Are you suggesting that only the "modules" subdirectory will actually be 
needed by users?  Anyway, the EB module building requires access to all 
subdirectories, so it seems convenient to have all subdirectories 
exposed this way.  Am I making any mistakes here?



The recommended line "module --initial_load restore" generates an
unwarranted message:


The system default contains no modules
  (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty)
  No changes in loaded modules


so I got rid of it by commenting this line out.


Well, you need to define $LMOD_SYSTEM_DEFAULT_MODULES :)


Do I really *need* to?  What's the point, and what should I do then?


Question: Does anyone see potential problems or bad side effects with
such a global setup of EB for all users?


Having a proper default setup in place makes sense, but users can easily
override it with their own config file or via environment variables.


That sounds like a "Good Thing" for advanced users, right?


The only thing we do is load Lmod itself (a symlink to init/bash in
/etc/profile.d).


Question: How do your users initialize the usage of EB?


They can define a config file, or define environment variables
(typically via .bashrc), up to them...


Ah, so it's DIY for your users!  I'm trying to make it easy for new 
users so that they don't need to manipulate .bashrc after reading some 
web documentation...  I guess that's a matter of taste.


/Ole

Re: [easybuild] no module for CUDA-7.5.18

2016-10-13 Thread Kenneth Hoste


Hi Joachim,

On 11/10/16 11:57, Joachim Hein wrote:

Hi,

We are still struggling to build a CUDA 7.5.18 with EB 2.9.0.  It 
seems to be failing in the “sanity check”, though the actual nvidia 
installer doesn’t give an error and the bin directory looks reasonable 
to us.  We get a “KeyError: ‘GCC’


Anyone could comment with regards to what went wrong:

-bash-4.2$ eb CUDA-7.5.18.eb
== temporary log file in case of crash /tmp/eb-wjA2JG/easybuild-2WrNil.log
== processing EasyBuild easyconfig 
/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.9.0-py2.7.egg/easybuild/easyconfigs/c/CUDA/CUDA-7.5.18.eb

== building and installing Core/CUDA/7.5.18...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
ERROR: Traceback (most recent call last):
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/main.py", 
line 115, in build_and_install_software
(ec_res['success'], app_log, err) = build_and_install_one(ec, 
init_env)
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py", 
line 2376, in build_and_install_one

result = app.run_all_steps(run_test_cases=run_test_cases)
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py", 
line 2292, in run_all_steps

self.run_step(step_name, step_methods)
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py", 
line 2171, in run_step

step_method(self)()
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_easyblocks-2.9.0-py2.7.egg/easybuild/easyblocks/c/cuda.py", 
line 137, in sanity_check_step

super(EB_CUDA, self).sanity_check_step(custom_paths=custom_paths)
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py", 
line 1790, in sanity_check_step

self._sanity_check_step(*args, **kwargs)
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py", 
line 1910, in _sanity_check_step

fake_mod_data = self.load_fake_module(purge=True)
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py", 
line 1183, in load_fake_module

fake_mod_path = self.make_module_step(fake=True)
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py", 
line 1997, in make_module_step

txt += self.make_module_extend_modpath()
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyblock.py", 
line 1072, in make_module_extend_modpath

modpath_exts = ActiveMNS().det_modpath_extensions(self.cfg)
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyconfig/easyconfig.py", 
line 1610, in det_modpath_extensions
modpath_extensions = 
self.mns.det_modpath_extensions(self.check_ec_type(ec))
  File 
"/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/tools/module_naming_scheme/hierarchical_mns.py", 
line 177, in det_modpath_extensions

comp_name_ver = [comp_name, comp_ver_tmpl % comp_versions]
KeyError: 'GCC'




You're running into a known issue, see 
https://github.com/hpcugent/easybuild-framework/pull/1753.


Basically, in the current implementation of HierarchicalMNS, installing 
CUDA at the 'Core' level is not supported.
You're running into an ugly traceback because of this, while PR #1753 at 
least throws a decent error message.


The idea was to enhance this further to also allow CUDA at the 'Core' 
level, but then the PR lost attention.

I'm happy to pick this up again and try and fix it...

Let's follow up in the PR?



regards,

Kenneth

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Kenneth Hoste




On 13/10/16 16:11, Ole Holm Nielsen wrote:

On 10/13/2016 03:28 PM, Ward Poelmans wrote:
For loading Easybuild by default, I would use a Lmod default 
collection.


Could you possibly offer a specific example of what you mean? Like a
complete script to put into /etc/profile.d/ ?


I'm afraid I don't have one. We don't load Easybuild by default for our
users. But you should take a look at:
https://lmod.readthedocs.io/en/latest/070_standard_modules.html


Thanks, this page is actually useful and specific!  So I've created a 
new file /etc/profile.d/z01_EasyBuild.sh (to be loaded *after* Lmod's 
z00_* scripts) with this content:


if [ -z "$__Init_Default_Modules" ]; then
   export __Init_Default_Modules=1;
   # module --initial_load restore
   export EASYBUILD_MODULES_TOOL=Lmod
   export EASYBUILD_PREFIX=/home/modules
   module use $EASYBUILD_PREFIX/modules/all

Is this right?

Are the modules really in /home/modules/modules/all?



   module load EasyBuild
else
   module refresh
fi

This seems to do what I want: Initialize EB with the specific usage of 
Lmod and our EASYBUILD_PREFIX.  N.B.: This is on CentOS 7.2.


The recommended line "module --initial_load restore" generates an 
unwarranted message:



The system default contains no modules
  (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty)
  No changes in loaded modules


so I got rid of it by commenting this line out.


Well, you need to define $LMOD_SYSTEM_DEFAULT_MODULES :)



Question: Does anyone see potential problems or bad side effects with 
such a global setup of EB for all users?


Having a proper default setup in place makes sense, but users can easily 
override it with their own config file or via environment variables.





The only thing we do is load Lmod itself (a symlink to init/bash in
/etc/profile.d).


Question: How do your users initialize the usage of EB?


They can define a config file, or define environment variables 
(typically via .bashrc), up to them...



K.

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Ole Holm Nielsen


On 10/13/2016 03:28 PM, Ward Poelmans wrote:

For loading Easybuild by default, I would use a Lmod default collection.


Could you possibly offer a specific example of what you mean?  Like a
complete script to put into /etc/profile.d/ ?


I'm afraid I don't have one. We don't load Easybuild by default for our
users. But you should take a look at:
https://lmod.readthedocs.io/en/latest/070_standard_modules.html


Thanks, this page is actually useful and specific!  So I've created a 
new file /etc/profile.d/z01_EasyBuild.sh (to be loaded *after* Lmod's 
z00_* scripts) with this content:


if [ -z "$__Init_Default_Modules" ]; then
   export __Init_Default_Modules=1;
   # module --initial_load restore
   export EASYBUILD_MODULES_TOOL=Lmod
   export EASYBUILD_PREFIX=/home/modules
   module use $EASYBUILD_PREFIX/modules/all
   module load EasyBuild
else
   module refresh
fi

This seems to do what I want: Initialize EB with the specific usage of 
Lmod and our EASYBUILD_PREFIX.  N.B.: This is on CentOS 7.2.


The recommended line "module --initial_load restore" generates an 
unwarranted message:



The system default contains no modules
  (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty)
  No changes in loaded modules


so I got rid of it by commenting this line out.

Question: Does anyone see potential problems or bad side effects with 
such a global setup of EB for all users?



The only thing we do is load Lmod itself (a symlink to init/bash in
/etc/profile.d).


Question: How do your users initialize the usage of EB?

/Ole

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Ward Poelmans

On 13-10-16 15:20, Ole Holm Nielsen wrote:
> On 10/13/2016 01:54 PM, Ward Poelmans wrote:
>> On 13-10-16 13:48, Ole Holm Nielsen wrote:
>>
>> For loading Easybuild by default, I would use a Lmod default collection.
> 
> Could you possibly offer a specific example of what you mean?  Like a
> complete script to put into /etc/profile.d/ ?

I'm afraid I don't have one. We don't load Easybuild by default for our
users. But you should take a look at:
https://lmod.readthedocs.io/en/latest/070_standard_modules.html

The only thing we do is load Lmod itself (a symlink to init/bash in
/etc/profile.d).

Ward

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Ole Holm Nielsen


On 10/13/2016 01:54 PM, Ward Poelmans wrote:

On 13-10-16 13:48, Ole Holm Nielsen wrote:

I would like to enable the EB/Lmod modules to all users in a global
bash/tcsh setup script.  On CentOS 7.2 initialization is done using
scripts in /etc/profile.d/

Basically I want every user shell to execute these commands:

export EASYBUILD_MODULES_TOOL=Lmod
export EASYBUILD_PREFIX=/home/modules
module use $EASYBUILD_PREFIX/modules/all
module load EasyBuild


The settings you can put in a global config file? Much safer then
environment variables.

For loading Easybuild by default, I would use a Lmod default collection.


Could you possibly offer a specific example of what you mean?  Like a 
complete script to put into /etc/profile.d/ ?


Thanks,
Ole

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Kenneth Hoste


Hi André,

On 13/10/16 14:39, André Gemünd wrote:

Hi Ward,

sorry for interfering with this thread, but *is* there actually a global config 
file? I asked this some time ago on the ml, and as far as I remember there are 
only local config files ($XDG_...), no global config e.g. under the 
installation prefix.


See 
http://easybuild.readthedocs.io/en/latest/Configuration.html#default-configuration-files 
(and the demo available at 
http://easybuild.readthedocs.io/en/latest/demos/configuring.html).


By default, EasyBuild will consider /etc/easybuild.d/*.cfg, but it's 
true you can override this via $XDG_CONFIG_DIRS .


If that's an issue, we can certainly consider having a hardcoded 
location in there that can not be tweaked, if this is desirable...




regards,

Kenneth


Cheers
Andre

- Am 13. Okt 2016 um 13:54 schrieb Ward Poelmans ward.poelm...@ugent.be:


On 13-10-16 13:48, Ole Holm Nielsen wrote:

I would like to enable the EB/Lmod modules to all users in a global
bash/tcsh setup script.  On CentOS 7.2 initialization is done using
scripts in /etc/profile.d/

Basically I want every user shell to execute these commands:

export EASYBUILD_MODULES_TOOL=Lmod
export EASYBUILD_PREFIX=/home/modules
module use $EASYBUILD_PREFIX/modules/all
module load EasyBuild

The settings you can put in a global config file? Much safer then
environment variables.

For loading Easybuild by default, I would use a Lmod default collection.

Ward

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread André Gemünd

Hi Ward,

sorry for interfering with this thread, but *is* there actually a global config 
file? I asked this some time ago on the ml, and as far as I remember there are 
only local config files ($XDG_...), no global config e.g. under the 
installation prefix.

Cheers
Andre

- Am 13. Okt 2016 um 13:54 schrieb Ward Poelmans ward.poelm...@ugent.be:

> On 13-10-16 13:48, Ole Holm Nielsen wrote:
>> I would like to enable the EB/Lmod modules to all users in a global
>> bash/tcsh setup script.  On CentOS 7.2 initialization is done using
>> scripts in /etc/profile.d/
>> 
>> Basically I want every user shell to execute these commands:
>> 
>> export EASYBUILD_MODULES_TOOL=Lmod
>> export EASYBUILD_PREFIX=/home/modules
>> module use $EASYBUILD_PREFIX/modules/all
>> module load EasyBuild
> 
> The settings you can put in a global config file? Much safer then
> environment variables.
> 
> For loading Easybuild by default, I would use a Lmod default collection.
> 
> Ward

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Riccardo Murri

> Question: Can anyone share their /etc/profile.d/ script for EB setup?

I use these in ElastiCluster (Jinja templates but should be still quite
readable); I set user-independent options in `/etc/easybuild.cfg` and
options that might need to be computed on a by-user basis in
`/etc/profile.d/easybuild.sh`::

https://github.com/gc3-uzh-ch/elasticluster/blob/master/elasticluster/share/playbooks/roles/easybuild/templates/etc/profile.d/easybuild.sh.j2
https://github.com/gc3-uzh-ch/elasticluster/blob/master/elasticluster/share/playbooks/roles/easybuild/templates/etc/easybuild.cfg.j2

To load EB for all users, ElastiCluster uses the "StdEnv" approach
advocated by Lmod's documentation (I guess this also what Ward was
referring to)::

https://github.com/gc3-uzh-ch/elasticluster/blob/master/elasticluster/share/playbooks/roles/lmod/templates/etc/profile.d/z80_StdEnv.sh.j2

Deployment of EB through ElastiCluster is still work in progress, but
the templates have been adapted from what we use on the local cluster so
are not very far from a working setup ;-)

Ciao,
R

-- 
Riccardo Murri
http://www.s3it.uzh.ch/about/team/#Riccardo.Murri

S3IT: Services and Support for Science IT
University of Zurich
Winterthurerstrasse 190, CH-8057 Zürich (Switzerland)

Tel: +41 44 635 4208
Fax: +41 44 635 6888

Re: [easybuild] FFTW dynamic libraries?

2016-10-13 Thread Kenneth Hoste


Hi Pablo,

On 13/10/16 14:06, Pablo Escobar Lopez wrote:

Hi Kenneth,

I needed this because while compiling an internal application I 
noticed that it requires "libfftw3f_threads.so". As my compute nodes 
have fftw-devel rpm installed the build system was using this library 
from the system. After recompiling my FFTW module with --enable-shared 
the application uses the libfftw3f_threads.so lib provided by my FFTW 
module instead of the system one.


These are the contents of my $EBROOTFFTW/lib/ folder when compiling 
with --enable-shared. It includes both the static and dynamic libraries:

https://gist.github.com/pescobar/90cf77b0bcb5c1494b18365ce50a78cb

IMHO, it makes sense that the FFTW installation in easybuild provides 
both the dynamic and static libraries so applications depending on the 
fftw dynamic libs can use the easbuild FFTW module. Does this have any 
drawback I am missing?


Note that I'm aware of, it's probably that nobody has needed the dynamic 
libraries specifically...


So, a PR to add --enable-shared would be nice. ;-)
Ideally, also with updating the sanity_check_paths...



K.


regards,
Pablo.



2016-10-13 12:47 GMT+02:00 Kenneth Hoste >:


Hi Pablo,

On 13/10/16 11:48, Pablo Escobar Lopez wrote:

Hi,

I have noticed that FFTW-3.3.3-gompi-1.4.10.eb is not building
the dynamic FFTW libraries. For this I think the fix would be to
change configopts from this:

common_configopts = "--enable-threads --enable-openmp --with-pic"

to

common_configopts = "--enable-threads --enable-openmp --with-pic
*_--enable-shared_*"

Is there any reason why the provided FFTW easyconfig is not
building the dynamic libraries. Would it be safe to apply this
change so the dynamic libs are built? or is there any issue which
could be triggered by doing this change which I am missing?


It's probably just because nobody has had a strict need for the
dynamic libraries yet? What's your use case for needing them?

The only thing I can think of is that by using --enable-shared the
static libraries may not be built anymore, but you'll have to
check that (and the sanity check should fail anyway in that case).


regards,

Kenneth




--
Pablo Escobar López
HPC systems engineer
sciCORE, University of Basel
SIB Swiss Institute of Bioinformatics
http://scicore.unibas.ch

Re: [easybuild] FFTW dynamic libraries?

2016-10-13 Thread Pablo Escobar Lopez

Hi Kenneth,

I needed this because while compiling an internal application I noticed
that it requires "libfftw3f_threads.so". As my compute nodes have
fftw-devel rpm installed the build system was using this library from the
system. After recompiling my FFTW module with --enable-shared the
application uses the libfftw3f_threads.so lib provided by my FFTW module
instead of the system one.

These are the contents of my $EBROOTFFTW/lib/ folder when compiling with
--enable-shared. It includes both the static and dynamic libraries:
https://gist.github.com/pescobar/90cf77b0bcb5c1494b18365ce50a78cb

IMHO, it makes sense that the FFTW installation in easybuild provides both
the dynamic and static libraries so applications depending on the fftw
dynamic libs can use the easbuild FFTW module. Does this have any drawback
I am missing?

regards,
Pablo.

2016-10-13 12:47 GMT+02:00 Kenneth Hoste :

> Hi Pablo,
>
> On 13/10/16 11:48, Pablo Escobar Lopez wrote:
>
> Hi,
>
> I have noticed that FFTW-3.3.3-gompi-1.4.10.eb is not building the
> dynamic FFTW libraries. For this I think the fix would be to change
> configopts from this:
>
> common_configopts = "--enable-threads --enable-openmp --with-pic"
>
> to
>
> common_configopts = "--enable-threads --enable-openmp --with-pic
> *--enable-shared*"
>
> Is there any reason why the provided FFTW easyconfig is not building the
> dynamic libraries. Would it be safe to apply this change so the dynamic
> libs are built? or is there any issue which could be triggered by doing
> this change which I am missing?
>
>
> It's probably just because nobody has had a strict need for the dynamic
> libraries yet? What's your use case for needing them?
>
> The only thing I can think of is that by using --enable-shared the static
> libraries may not be built anymore, but you'll have to check that (and the
> sanity check should fail anyway in that case).
>
>
> regards,
>
> Kenneth
>
>

-- 
Pablo Escobar López
HPC systems engineer
sciCORE, University of Basel
SIB Swiss Institute of Bioinformatics
http://scicore.unibas.ch

Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Ward Poelmans

On 13-10-16 13:48, Ole Holm Nielsen wrote:
> I would like to enable the EB/Lmod modules to all users in a global
> bash/tcsh setup script.  On CentOS 7.2 initialization is done using
> scripts in /etc/profile.d/
> 
> Basically I want every user shell to execute these commands:
> 
> export EASYBUILD_MODULES_TOOL=Lmod
> export EASYBUILD_PREFIX=/home/modules
> module use $EASYBUILD_PREFIX/modules/all
> module load EasyBuild

The settings you can put in a global config file? Much safer then
environment variables.

For loading Easybuild by default, I would use a Lmod default collection.

Ward

[easybuild] How to setup EB in the global /etc/profile.d/ ?

2016-10-13 Thread Ole Holm Nielsen

I would like to enable the EB/Lmod modules to all users in a global 
bash/tcsh setup script.  On CentOS 7.2 initialization is done using 
scripts in /etc/profile.d/


Basically I want every user shell to execute these commands:

export EASYBUILD_MODULES_TOOL=Lmod
export EASYBUILD_PREFIX=/home/modules
module use $EASYBUILD_PREFIX/modules/all
module load EasyBuild

Question: Can anyone share their /etc/profile.d/ script for EB setup?

Some sanity checks must be performed, among other things the Lmod RPM 
package has created these scripts:


$ ls -la /etc/profile.d/*mod*
-rw-r--r--. 1 root root 122 May 12 23:57 /etc/profile.d/00-modulepath.csh
-rw-r--r--. 1 root root 113 May 12 23:57 /etc/profile.d/00-modulepath.sh
lrwxrwxrwx. 1 root root  31 Jul 18 15:58 /etc/profile.d/z00_lmod.csh -> 
/usr/share/lmod/lmod/init/cshrc
lrwxrwxrwx. 1 root root  33 Jul 18 15:58 /etc/profile.d/z00_lmod.sh -> 
/usr/share/lmod/lmod/init/profile


The z00_lmod.* script should be performed prior to the "module" commands.

Thanks for sharing any good ideas.

/Ole

Re: [easybuild] FFTW dynamic libraries?

2016-10-13 Thread Kenneth Hoste


Hi Pablo,

On 13/10/16 11:48, Pablo Escobar Lopez wrote:

Hi,

I have noticed that FFTW-3.3.3-gompi-1.4.10.eb is not building the 
dynamic FFTW libraries. For this I think the fix would be to change 
configopts from this:


common_configopts = "--enable-threads --enable-openmp --with-pic"

to

common_configopts = "--enable-threads --enable-openmp --with-pic 
*_--enable-shared_*"


Is there any reason why the provided FFTW easyconfig is not building 
the dynamic libraries. Would it be safe to apply this change so the 
dynamic libs are built? or is there any issue which could be triggered 
by doing this change which I am missing?


It's probably just because nobody has had a strict need for the dynamic 
libraries yet? What's your use case for needing them?


The only thing I can think of is that by using --enable-shared the 
static libraries may not be built anymore, but you'll have to check that 
(and the sanity check should fail anyway in that case).



regards,

Kenneth

Re: [easybuild] matplotlib installation fails with ImportError

2016-10-13 Thread Kenneth Hoste

Hi Maik,

I just tried to reinstall matplotlib-1.5.1-intel-2016a-Python-3.5.1.eb ,
and I'm not seeing that a newer numpy is being picked up at all...