Hi,
I am administrating a few dozen heterogeneous compute nodes running CentOS 6.7
64-bit and have been struggling to bend the new build system of FEniCS to our
systems' needs since 1.5.0.
I had been able to tweak the FEniCS 1.4.0 build system to
* not compile ATLAS BLAS,
* not compile Python,
* not compile Boost,
* not compile OpenMPI,
* use a newer CGAL and
* have all FEniCS prerequisites use the OpenBLAS installation in a non-default
path.
None of this I have managed with neither 1.5.0 nor 1.6.0. In particular trying
to avoid to rebuild Python has proven difficult. I have skimmed over and grepped
through the mailing list archive to no avail which is why I am now resorting to
contact you.
The system heavily relies on Environment Modules and I am trying to avoid that
any of the following modules get rebuilt for the sole use of FEniCS 1.6.0. In
particular, OpenBLAS must not be compiled by FEniCS itself to avoid have a
non-relocatable installation (not so much path-relocatable, but in the sense of
rsync'able to other servers with different CPUs and cache sizes where an
appropriate OpenBLAS installation is readily available already):
-------------------- /sfw/Modules/modulefiles ---------------------
cmake/2.8.12 cmake/3.2.2
boost/1.41-centos boost/1.48 boost/1.56
gcc/4.4.7-centos gcc/4.9.2
gcc/4.7.3 gcc/5.1.0
gcc/4.8.5 gcc/5.2.0
openmpi/gcc4.4.x/1.8.2 openmpi/intel13.1.x/1.8.5
openmpi/gcc4.4.x/1.8.5 openmpi/intel14.0.x/1.8.5
openmpi/gcc4.4.x-ib-gnu/1.8.5 openmpi/intel15.0.x/1.8.5
openmpi/gcc4.9.x/1.8.5 openmpi/intel15.0.x-ib-gpu/1.10.1
openmpi/gcc5.1.x/1.8.5
openblas/0.2.13(deprec) openblas/0.2.15
python/2.7.8
I have already customized a *.yaml script based on what I could find on the
mailing list. I am wondering whether you could give me some pointers as to where
I set up the script attached wrongly. (I run it via 'bash; source
install-script.wip') For the last two days, I am stuck at the FEniCS 1.6.0
installation trying to build python - although I thought I had forbidden that in
install-script.wip - and failing to locate /usr/bin/python-config (which does
indeed not exist there, but in /sfw/python/2.7.8/bin/python-config)
Thank you in advance for taking the time to answer me and possibly provide me
with hints.
With kind regards,
Sven Buijssen
--
Dipl.Math. Sven H.M. Buijssen
wiss.Ang./MRB/LiDO-Team
Technische Universität Dortmund
Institut für Angewandte Mathematik (LS III)
Fakultät für Mathematik
Vogelpothsweg 87
D-44227 Dortmund
Tel.: +49 231-755 59 34
Fax: +49 231-755 59 33
[email protected]
www.tu-dortmund.de www.mathematik.tu-dortmund.de/lsiii
www.mathematik.tu-dortmund.de/~buijssen
Important note: The information included in this e-mail is confidential. It is
solely intended for the recipient. If you are not the intended recipient of this
e-mail please contact the sender and delete this message. Thank you.
Without prejudice of e-mail correspondence, our statements are only legally
binding when they are made in the conventional written form (with personal
signature) or when such documents are sent by fax.
# FEniCS 1.6.0 build tool is hashdist, which is controlled by several, nested
# *.yaml files. Problem is that FEniCS tries to build everything. Even if it
# already is available: m4, bzip2, openblas, cmake, python, boost. And only for
# the current machine. Such that the build is not relocatable to different
# hardware (where we already do provide a different OpenBLAs etc.)
#
# On top of that, the build tool overwrites PATH with /usr/bin:/bin,
# effectively ignoring any loaded modulefile.
# SBtmp ssh whitewalker
# SBtmp yum install libXmu-devel
# SBtmp cd /scratch/$USER
module purge
# SBtmp # Make sure that python modulefile sets PYTHON_INCLUDE_DIR and
PYTHON_LIBRARY to
# SBtmp # to kickstart the detection by VTK in FindPython*.cmake!
module load python/2.7.8
# SBtmp # Do *not* use cmake/3.2.2, probably no cmake/3* is suitable
# SBtmp # You will get many COPY_ONLY warnings during CMake configures
# SBtmp # and eventually PetSC will fail to compile against ParMetis.
# SBtmp module load openmpi/gcc4.9.x/1.8.5 cmake/2.8.12 boost/1.56
openblas/0.2.15 gcc/4.9.2
# SBtmp sudo chown $USER /sfw/fenics
curl http://fenicsproject.org/fenics-install.sh --output fenics-install.sh
sed -i -e "s,^\(DIR=..mktemp -d \)/tmp,\1/scratch/$USER," fenics-install.sh
mkdir ~/.hashdist
cat <<EOF > ~/.hashdist/config.yaml
## Configuration file for HashDist
## All relative paths are relative to the directory containing this
## configuration file.
## Where to store the built software
build_stores:
- dir: /sfw/fenics/1.6.0
# - dir: $HOME/fenics/1.6.0
## Location where temporary directories for building software are created.
## Such directories are by default removed again once the build is done.
build_temp: /scratch/$USER/tmp
## Locations of downloaded tarballs and git repositories. A location
## can either be a local filesystem, or a URL to an online, read-only
## mirror. Only the first location, which should be a local directory,
## will be written to.
source_caches:
- dir: /scratch/$USER/src
## For additional source cache mirror:
## - url: https://some.server.org/hashdist/src
## The cache directory is used for misc. caching (e.g., probing of host
## system). The contents can always be wiped without resulting in rebuilds.
cache: /scratch/$USER/cache
## The roots directory contains links to (links to) profiles. Anything
## pointed to through here will not be deleted when garbage-collected.
gc_roots: /scratch/$USER/gcroots
EOF
export FENICS_INSTALL_HASHDIST_PROFILE=/scratch/$USER/fenics.mrb.yaml
export HASHDIST_PROFILE=$FENICS_INSTALL_HASHDIST_PROFILE
#export FENICS_INSTALL_BUILD_TYPE=custom
export FENICS_INSTALL_BUILD_TYPE=1
cat <<EOF > $FENICS_INSTALL_HASHDIST_PROFILE
# settings based on ...
# ... fenics-install.*/hashstack/config.yaml (downloaded when fenics-install.sh
is run)
package_dirs:
- pkgs
- base
hook_import_dirs:
- base
parameters:
BASH: /bin/bash
HOST_MPICC: /sfw/openmpi/gcc4.9.x/1.8.5/bin/mpicc
HOST_MPICXX: /sfw/openmpi/gcc4.9.x/1.8.5/bin/mpic++
HOST_MPIF77: /sfw/openmpi/gcc4.9.x/1.8.5/bin/mpif77
HOST_MPIF90: /sfw/openmpi/gcc4.9.x/1.8.5/bin/mpif90
HOST_MPIEXEC: /sfw/openmpi/gcc4.9.x/1.8.5/bin/mpiexec
HOST_BLAS_DIR: /usr/local/lib/openblas/current
HOST_LAPACK: /usr/local/lib/openblas/current
HOST_CMAKE: /sfw/cmake/2.8.12/bin/cmake
HOST_PYTHON: /sfw/python/2.7.8/bin/python
PYTHON: /sfw/python/2.7.8/bin/python
PYTHON_DIR: /sfw/python/2.7.8
# ----
# ... fenics-install.*/hashstack/linux.yaml
platform: linux
fortran: true
#PATH: /usr/bin:/bin
PATH:
/sfw/gcc/4.9.2-static-gmp-mpfr-mpc-isl-cloog/bin:/sfw/openmpi/gcc4.9.x/1.8.5/bin:/sfw/cmake/2.8.12/bin:/sfw/python/2.7.8/bin:/usr/bin:/bin
PROLOGUE: | # by default, PROLOGUE is a no-op
source /sfw/Modules/default/init/bash || true; echo "loading modules";
module use /sfw/Modules/modulefiles; module load gcc/4.9.2; module load
openmpi/gcc4.9.x/1.8.5; module load cmake/2.8.12; module load boost/1.56;
module load openblas/0.2.15; echo "setting default compilers"; export CC=gcc;
export CXX=g++; export FC=gfortran; export F77=gfortran; export CPP=cpp;
# Note, if you change pyver,
# you should probably change the following two lines as well.
# pyver: '2.7' # as in used in lib/pythonX.Y/site-packages
python_site_packages_rel: 'lib/python2.7/site-packages'
# python_host_packages: '/sfw/python/2.7.8'
# use_python_host_packages: true
# ----
# ... fenics-install.*/fenics-developer-tools/install/profiles/fenics.Linux.yaml
# The following debian packages should be installed when using this
# profile:
#
# git curl python-dev time build-essential gfortran freeglut3-dev
# This profile file controls your <#> (HashDist) build environment.
# In the future, we'll provide better incorporation of
# automatic environment detection. For now, have a look
# at the YAML files in the top-level directory and choose
# the most *specific* file that matches your environment.
#extends:
#- file: linux.yaml
# The packages list specifies all the packages that you
# require installed. <#> will ensure that all packages
# and their dependencies are installed when you build this
# profile.
packages:
launcher:
# cmake:
# python:
# link: shared
# mpi:
# use: mpich
# blas:
# use: openblas
# lapack:
# use: openblas
hypre:
with_openblas: true
without_check: true
petsc:
version: '3.6.1'
build_with: |
openblas, parmetis, scotch, suitesparse, superlu_dist, hypre
download: |
mumps, scalapack, blacs, ml
coptflags: -O2
link: shared
debug: false
petsc4py:
version: '3.6.0'
slepc:
version: '3.6.1'
slepc4py:
version: '3.6.0'
swig:
# boost:
# toolset: gcc
# build_with: python
ipython:
matplotlib:
ffc:
fiat:
instant:
ufl:
uflacs:
dolfin:
build_with: |
openblas, hdf5, parmetis, petsc, petsc4py, suitesparse, scotch, slepc,
slepc4py, vtk, zlib
mpi4py:
mshr:
cmake:
use: host-cmake
mpi:
use: host-mpi
python:
host: true
# build_with: |
# openssl, bzip2, sqlite
# link: shared
# use_python_host_packages: true
boost:
use: host-boost
blas:
use: host-blas
lapack:
use: host-lapack
EOF
bash fenics-install.sh
2015/11/10 12:20:23 - INFO: [package:run_job] running [u'/bin/bash', '_hashdist/build.sh']
2015/11/10 12:20:23 - INFO: [package:run_job] environment:
2015/11/10 12:20:23 - INFO: [package:run_job] {'ARTIFACT': u'/sfw/fenics/1.6.0/python/w7kg7ww4ffez',
2015/11/10 12:20:23 - INFO: [package:run_job] 'BASH': u'/bin/bash',
2015/11/10 12:20:23 - INFO: [package:run_job] 'BUILD': u'/export/data/scratch/buijssen/tmp/python-w7kg7ww4ffez-2',
2015/11/10 12:20:23 - INFO: [package:run_job] 'HASHDIST_CPU_COUNT': '4',
2015/11/10 12:20:23 - INFO: [package:run_job] 'HDIST_CONFIG': '{"gc_roots":"/scratch/buijssen/gcroots","build_stores":[{"dir":"/sfw/fenics/1.6.0"}],"source_caches":[{"dir":"/scratch/buijssen/src"}],"cache":"/scratch/buijssen/cache","build_temp":"/scratch/buijssen/tmp"}',
2015/11/10 12:20:23 - INFO: [package:run_job] 'HDIST_IMPORT': '',
2015/11/10 12:20:23 - INFO: [package:run_job] 'HDIST_IMPORT_PATHS': '',
2015/11/10 12:20:23 - INFO: [package:run_job] 'HDIST_VIRTUALS': '',
2015/11/10 12:20:23 - INFO: [package:run_job] 'PATH': u'/sfw/gcc/4.9.2-static-gmp-mpfr-mpc-isl-cloog/bin:/sfw/openmpi/gcc4.9.x/1.8.5/bin:/sfw/cmake/2.8.12/bin:/sfw/python/2.7.8/bin:/usr/bin:/bin',
2015/11/10 12:20:23 - INFO: [package:run_job] 'PWD': u'/export/data/scratch/buijssen/tmp/python-w7kg7ww4ffez-2'}
2015/11/10 12:20:23 - INFO: [package:run_job] cp: cannot stat `/usr/bin/python-config': No such file or directory
2015/11/10 12:20:23 - ERROR: [package:run_job] Command '[u'/bin/bash', '_hashdist/build.sh']' returned non-zero exit status 1
2015/11/10 12:20:23 - ERROR: [package:run_job] command failed (code=1); raising
_______________________________________________
fenics-support mailing list
[email protected]
http://fenicsproject.org/mailman/listinfo/fenics-support
