Re: [easybuild] Building R with GDAL and sf
Hi John,
On 06/11/2019 23:24, Dey, John F wrote:
Has anyone added ‘sf’ to R with GDAL 3?
I’m doing this; but still get an error about -std=c++11 required. I’ve
also tried using configopts,
('sf', '0.8-0', {
'buildopts': 'CXXFLAGS="$CXXFLAGS -std=c++11"'
}),
That won't work as expected with R CMD INSTALL, you need to use
--configure-args="...", see
https://stackoverflow.com/questions/10921153/c-compilation-flags-from-r .
Try this:
('sf', '0.8-0', {
'installopts': '--configure-args="CXXFLAGS=\\"$CXXFLAGS
-std=c++11\\""',
}),
You need to make sure you're using double quotes so the $CXXFLAGS is
expanded rather than being passed literally, and to install "$CXXFLAGS
-std=c++11" as a whole to CXXFLAGS=
regards,
Kenneth
== 2019-11-06 13:50:59,986 build_log.py:163 ERROR EasyBuild crashed with
an error (at
easybuild/software/EasyBuild/3.9.2/lib/python2.7/site-packages/vsc_base-2.8.4-py2.7.egg/vsc/utils/exceptions.py:124
in __init__): cmd " R CMD INSTALL
/app/easybuild/sources/r/R/extensions/sf_0.8-0.tar.gz
--library=/app/easybuild/software/R/3.6.1-foss-2016b-monocle3/lib/R/library
--no-clean-on-error " exited with exit code 1 and output:
* installing *source* package ‘sf’ ...
** package ‘sf’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: gcc
configure: CXX: g++
checking for gdal-config...
/app/easybuild/software/GDAL/3.0.0-foss-2016b-Python-3.7.4/bin/gdal-config
checking gdal-config usability... yes
configure: GDAL: 3.0.0
checking GDAL version >= 2.0.1... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking gdal.h usability... yes
checking gdal.h presence... yes
checking for gdal.h... yes
checking GDAL: linking with --libs only... no
checking GDAL: linking with --libs and --dep-libs... no
In file included from
/app/easybuild/software/GDAL/3.0.0-foss-2016b-Python-3.7.4/include/gdal.h:45:0,
from gdal_test.cpp:1:
/app/easybuild/software/GDAL/3.0.0-foss-2016b-Python-3.7.4/include/cpl_port.h:187:6:
error: #error Must have C++11 or newer.
# error Must have C++11 or newer.
^
https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/r/RQGIS3/RQGIS3-20190903-foss-2019a-R-3.6.0.eb
*John Dey*
*HPC Operations*
SciComp Computing
*O* _206.667.4308_
*M* _360.649.2731_
jf...@fredhutch.org
SciComp Office Hours
Wed. 10-noon M4-B102
Re: [easybuild] EasyBuild in a heterogeneous HPC centre
Hi Douglas, For our Linux cluster with 4 generations of hardware, I chose to make completely separate module trees for each type. IMHO, this is the simplest and most easily maintainable solution, although it is slightly wasteful of storage space on our central NFS server. It has worked well for several years now. I documented my approach in a Wiki page: https://wiki.fysik.dtu.dk/niflheim/EasyBuild_modules#automounting-the-cpu-architecture-dependent-modules-directory Best regards, Ole On 11/7/19 12:57 PM, Douglas Scofield wrote: We are curious how to maintain architecture-specific EasyBuild trees. We are new to EasyBuild and already have many, many (mostly bioinformatics) tools installed in hand-maintained module and software trees. In our centre, we have clusters running Sandy Bridge EP, Haswell EP, and Broadwell EP. Most of our users are on Broadwell, so we if we compile with -march=native etc., we compile for Broadwell and for Sandy Bridge, which covers it and Haswell. In our standard module tree we manage architecture automatically, keying off a $Cluster variable set by the module system. We handle architectures as if we had modules versioned Tool/Version/Architecture, with the last bit hidden from the user. We have also decided to (mostly) hide our EasyBuild tree from the user, and instead provide access to EasyBuild-built tools using what we are calling alias modules, which we place in our standard module tree. An alias module performs a 'module use' of the easybuild tree and then loads the appropriate EasyBuild-built modules. The large majority of our users do not care about toolchains, etc. Those that do, we will have docs they can consult for working with EasyBuild modules directly. The very large majority of our installed tools do not currently have easyconfigs. We are guessing that our architecture solution with EasyBuild will end up being completely separate EasyBuild trees, accessed using distinct 'module use' paths. The EasyBuild docs point to a 2015 presentation by Pablo Escobar describing an automounter solution which we are definitely not going to implement, but this suggests completely separate trees as well. How is this typically handled by other centres ? Thanks in advance, Douglas — Douglas G. Scofield Evolutionary Biology Centre, Uppsala University douglas.scofi...@ebc.uu.se douglasgscofi...@gmail.com När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/ E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark, Fysikvej Building 309, DK-2800 Kongens Lyngby, Denmark E-mail: ole.h.niel...@fysik.dtu.dk Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/ Mobile: (+45) 5180 1620
[easybuild] EasyBuild in a heterogeneous HPC centre
Hej, We are curious how to maintain architecture-specific EasyBuild trees. We are new to EasyBuild and already have many, many (mostly bioinformatics) tools installed in hand-maintained module and software trees. In our centre, we have clusters running Sandy Bridge EP, Haswell EP, and Broadwell EP. Most of our users are on Broadwell, so we if we compile with -march=native etc., we compile for Broadwell and for Sandy Bridge, which covers it and Haswell. In our standard module tree we manage architecture automatically, keying off a $Cluster variable set by the module system. We handle architectures as if we had modules versioned Tool/Version/Architecture, with the last bit hidden from the user. We have also decided to (mostly) hide our EasyBuild tree from the user, and instead provide access to EasyBuild-built tools using what we are calling alias modules, which we place in our standard module tree. An alias module performs a 'module use' of the easybuild tree and then loads the appropriate EasyBuild-built modules. The large majority of our users do not care about toolchains, etc. Those that do, we will have docs they can consult for working with EasyBuild modules directly. The very large majority of our installed tools do not currently have easyconfigs. We are guessing that our architecture solution with EasyBuild will end up being completely separate EasyBuild trees, accessed using distinct 'module use' paths. The EasyBuild docs point to a 2015 presentation by Pablo Escobar describing an automounter solution which we are definitely not going to implement, but this suggests completely separate trees as well. How is this typically handled by other centres ? Thanks in advance, Douglas — Douglas G. Scofield Evolutionary Biology Centre, Uppsala University douglas.scofi...@ebc.uu.se douglasgscofi...@gmail.com När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/ E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy
Re: [easybuild] Rust applications and libraries in eb?
Hi Christian, On 07/11/2019 10:20, Christian Meesters wrote: Hi, The recent updates of easybuild brought forward the Rust compiler as a easyconfig. Is there anyone build modules for applications with it? I would like to build a library and an application. For this: Would it be a good idea to have something similar to toolchain/compiler/pgi.py? Or is some more intricate approach involving a genuine toolchain better? Any ideas / remarks on this? Anyone already did some work, which did not find the way in the eb trunk? We already have a couple of easyconfigs for software implemented in Rust, see for example: https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/l/Longshot/Longshot-0.3.4-GCCcore-8.2.0.eb and https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/s/smafa/smafa-0.4.0-GCCcore-8.2.0.eb I don't think there's a need for introducing a custom toolchain for Rust itself, I see it more as being on the same level as Python & Perl (even though it's not interpreted). Just include Rust as a dependendy, and use the toolchain that was used to build Rust with, like we did in the examples. For toolchains, EasyBuild does more than just loading the corresponding module, it also sets up the build environment with $CC, $CFLAGS, etc. If you'd introduce a Rust-based toolchain, you'd have to do something equivalent for the Rust compiler, and that may involve more than just a toolchain/compiler/rust.py module (since currently the EasyBuild framework is mostly expecting C/C++/Fortran toolchain compilers). regards, Kenneth
Re: [easybuild] easybuild 4.0.1 doesn't work with env-modules 3.2.10 ?
Dear Sajid,
On 06/11/2019 20:41, Sajid Ali wrote:
Hi easy-builders,
I installed easybuild 4.0.1 by running the bootstrap_eb.py script and
tried to check if it was working by asking it to perform a dry run for
installation of python-3.8. I’m not why it doesn’t recognize the
modules-3.2.10 available on the system.
|[sajid@xrm temp]$ eb --version This is EasyBuild 4.0.1 (framework:
4.0.1, easyblocks: 4.0.1) on host xrm. [sajid@xrm temp]$ eb
Python-3.7.0-foss-2018b.eb -D == temporary log file in case of crash
/tmp/eb-7OETWx/easybuild-S3kK7q.log ERROR: Lmod modules tool can not be
used, 'lmod' command is not available; use --modules-tool to specify a
different modules tool to use (EnvironmentModulesTcl, EnvironmentMo
dulesC, EnvironmentModules, Lmod) [sajid@xrm temp]$ eb --modules-tool
EnvironmentModules ERROR: Failed to parse configuration options:
'Generating Lua module files requires Lmod as modules tool; use
--module-syntax to specify a different module syntax to use (Lua , Tcl)'
[sajid@xrm temp]$ eb --modules-tool EnvironmentModules --module-syntax
Tcl == temporary log file in case of crash
/tmp/eb-gpO28C/easybuild-qz72DR.log ERROR: EnvironmentModules modules
tool can not be used, '/usr/share/Modules/libexec/modulecmd.tcl' command
is not available; use --modules-tool to specify a different modules tool
to use (EnvironmentModulesTcl, EnvironmentModulesC, EnvironmentModules,
Lmod) [sajid@xrm temp]$ |
Looking for installed versions (of required eb dependencies) on the
workstation (running RHEL 7.4) :
|[sajid@xrm temp]$ python -V Python 2.7.5 [sajid@xrm temp]$ type module
module is a function module () { eval `/usr/bin/modulecmd bash $*` }
[sajid@xrm temp]$ type -f module -bash: type: module: not found
[sajid@xrm temp]$ module --version VERSION=3.2.10 DATE=2012-12-21
AUTOLOADPATH=undef BASEPREFIX="/usr/share" BEGINENV=99 CACHE_AVAIL=undef
DEF_COLLATE_BY_NUMBER=undef DOT_EXT="" EVAL_ALIAS=1 HAS_BOURNE_FUNCS=1
HAS_BOURNE_ALIAS=1 HAS_TCLXLIBS=undef HAS_X11LIBS=1 LMSPLIT_SIZE=1000
MODULEPATH="/etc/modulefiles" MODULES_INIT_DIR="/usr/share/Modules/init"
PREFIX="/usr/share/Modules" TCL_VERSION="8.5" TCL_PATCH_LEVEL="8.5.13"
TMP_DIR="/tmp" USE_FREE=undef VERSION_MAGIC=1 VERSIONPATH=undef
WANTS_VERSIONING=0 WITH_DEBUG_INFO=undef [sajid@xrm temp]$ which
modulecmd /bin/modulecmd [sajid@xrm temp]$ module av EasyBuild
---
/raid/home/sajid/.local/easybuild/modules/all
EasyBuild/4.0.1
[sajid@xrm temp]$ which -a eb
~/.local/easybuild/software/EasyBuild/4.0.1/bin/eb [sajid@xrm temp]$ eb
--version This is EasyBuild 4.0.1 (framework: 4.0.1, easyblocks: 4.0.1)
on host xrm. [sajid@xrm temp]$ |
Modules-3.2.10 is present and according to the docs it should work but
it doesn’t . What am i missing ?
To instruct EasyBuild to use Environment Modules 3.2.10 (which is
implemented with a mix of C and Tcl), you should configure EasyBuild to
use 'EnvirontmentModulesC' as modules tool.
The 'EnvironmentModules' value you used is for the modern 4.x versions
of Environment Modules (which is Tcl-only), while the
'EnvironmentModulesTcl' variant is for the ancient Tcl-only version.
So, to configure EasyBuild via environment variables:
export EASYBUILD_MODULES_TOOL=EnvironmentModulesC
export EASYBUILD_MODULE_SYNTAX=Tcl
I just noticed this isn't correctly documented at
https://easybuild.readthedocs.io/en/latest/Configuration.html, I'll get
that fixed...
I’m attaching the logfile from unit testing :
|[sajid@xrm temp]$ export
TEST_EASYBUILD_MODULES_TOOL=EnvironmentModulesC [sajid@xrm temp]$ python
-m test.framework.suite &> test.log |
Here you've correctly instructed the tests to use Environment Modules
3.2.10 using 'EnvironmentModulesC', but you also need to specify that
Tcl should be used as module syntax (the default is Lua, which only
works with Lmod):
export TEST_EASYBUILD_MODULES_TOOL=EnvironmentModulesC
export TEST_EASYBUILD_MODULE_SYNTAX=Tcl
Thanks in advance for the help!
regards,
Kenneth
[easybuild] Rust applications and libraries in eb?
Hi, The recent updates of easybuild brought forward the Rust compiler as a easyconfig. Is there anyone build modules for applications with it? I would like to build a library and an application. For this: Would it be a good idea to have something similar to toolchain/compiler/pgi.py? Or is some more intricate approach involving a genuine toolchain better? Any ideas / remarks on this? Anyone already did some work, which did not find the way in the eb trunk? Best regards, Christian
