Hi Baker,
Of course. Reposting to lmod and easybuild lists, since I suspect this
may be interesting to both lists
The first thing. All of our python modules add the same PYTHONPATH:
[mboisson@build-node ~]$ module show python/2.7.14 | grep PYTHONPATH
prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages")
[mboisson@build-node ~]$ module show python/3.5 | grep PYTHONPATH
prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages")
[mboisson@build-node ~]$ module show python/3.6 | grep PYTHONPATH
prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages")
In this PYTHONPATH, there is a single file : sitecustomize.py
This is a file that is always read by any python invokation (see the
official documentation https://docs.python.org/2/library/site.html
https://docs.python.org/3/library/site.html )
This file contains various things in our case, but the important part is
the following:
import sys
import os
import site
# use prefixes from EBPYTHONPREFIXES, so they have lower priority than
# virtualenv-installed packages
if "EBPYTHONPREFIXES" in os.environ:
postfix = os.path.join('lib', 'python'+sys.version[:3],
'site-packages')
for prefix in os.environ["EBPYTHONPREFIXES"].split(os.pathsep):
sitedir = os.path.join(prefix, postfix)
if os.path.isdir(sitedir):
site.addsitedir(sitedir)
As you can see from the logic, this means that, depending on the version
of python which is called, it will add a different path to the site
directories (through site.addsitedir) in which python will search for
modules.
This is then used to make python bindings work automagically for
software which do have such bindings. For example:
[mboisson@build-node ~]$ module show geos | grep "EBPYTHONPREFIXES\|python"
prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1")
[mboisson@build-node ~]$ ls -lhd
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1/lib/python*
drwxr-sr-x 3 ebuser ebuser 4,0K 4 déc 14:33
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1/lib/python2.7
[mboisson@build-node ~]$ module show igraph | grep
"EBPYTHONPREFIXES\|python"
prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1")
[mboisson@build-node ~]$ ls -lhd
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python*
drwxr-sr-x 3 ebuser ebuser 4,0K 26 sep 03:35
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python2.7
drwxr-sr-x 3 ebuser ebuser 4,0K 26 sep 03:35
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python3.5
In these examples, the python bindings for geos will work if you use
python2.7, and the igraph bindings will work if you use either python2.7
or python3.5. Note that we do not add any dependency of these modules
over any version of python. If a user loads python, the bindings will
work, if they don't load python, they don't need them to work.
For the few "pure python" modules that we offer (scipy-stack, which is
defined here https://www.scipy.org/stackspec.html and mpi4py because
different versions need to be provided depending on the MPI version that
is loaded), we also use this to make the same module work with any
version of python:
[mboisson@build-node ~]$ module show scipy-stack | grep
"EBPYTHONPREFIXES\|python\""
prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b")
depends_on("python")
[mboisson@build-node ~]$ ls -lhd
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python*
drwxr-sr-x 3 ebuser ebuser 4,0K 4 déc 22:33
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python2.7
drwxr-sr-x 3 ebuser ebuser 4,0K 4 déc 22:33
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python3.5
drwxr-sr-x 3 ebuser ebuser 4,0K 4 déc 22:34
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python3.6
[mboisson@build-node ~]$ module show mpi4py | grep
"EBPYTHONPREFIXES\|python\""
prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0")
depends_on("python")
[mboisson@build-node ~]$ ls -lhd
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python*
drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:31
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python2.7
drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:33
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python3.5
drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:34
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python3.6
In this case, since those modules are only useful for python, we do add
a "depends_on("python")" marker, but we do not specify any version. If a
user already had python loaded, it will work with that one. If they had
not, it will load our default version.
I hope this is useful.
Maxime
On 18-02-15 11:36, Baker D.J. wrote:
Hi Maxime,
Thank you again for your email, sent via the lmod mailing list,
regarding dealing with python modules. It would be really useful if
you could please elaborate on your python setup. I’ve got the basic
idea that the sitecustomize.py file is generating a new “search path”
depending upon the version of python being used – the prefix relating
to the packages and the postfix relating to the python version. Do you
have a concrete example that you could share please?
Furthermore, I understand that the scipy-stack is a software stack
providing python2/3 and a host of modules (numpy, scripy, etc) and
there would be a module for this stack. Given that, how do I select
whether I need to use python 2 or 3 in this setup?
Your clarification would be really appreciated, please.
Best regards,
David
*From:*Maxime Boissonneault [mailto:maxime.boissonnea...@calculquebec.ca]
*Sent:* Friday, February 02, 2018 9:14 PM
*To:* Baker D.J. <d.j.ba...@soton.ac.uk>; lmod-us...@lists.sourceforge.net
*Subject:* Re: [Lmod-users] lmod module sectioning & hierarchies --
dealing with python modules
Hi David,
My advice regarding python modules : don't. Python already has very
nice handling of dependencies, don't try to reproduce them through
modules. Instead, educate your users to use pip and virtualenv to
install whatever they need with their dependencies.
Most python packages these days have a binary version that is very
straightforward to install. If you prefer to build them for
performance reason, create a python "wheel house" instead.
https://wheel.readthedocs.io/en/stable/
A wheel house is simply a directory on your cluster in which you hosts
binary wheels that you will have compiled for your users.
Using $PIP_CONFIG_FILE, you can configure pip so that it first looks
into your wheel house first, and looks online only if what it needs
isn't found in the wheel house.
That is what we do on our cluster, and it works very very well.
The only python modules that we have are bundles (scipy-stack,
mpi4py). For those also, we do not use PYTHONPATH. PYTHONPATH is very
intrusive, and only works for one version of python or the other, even
though python is designed to be able to handle both python2 and
python3 at the same time since packages get installed in
pythonX.Y/site-packages/. Instead of using PYTHONPATH, we define a
difference environment variable (EBPYTHONPREFIXES), and we have a
sitecustomize.py file which looks at this directory, and will add
paths internally based on the version of python that is used :
postfix = os.path.join('lib', 'python'+sys.version[:3],
'site-packages')
nixstoresitepackages = os.path.join(sys.prefix, postfix)
if nixstoresitepackages in sys.path:
site.addsitedir(os.path.join(newprefix, postfix))
That way, our "scipy-stack" module works just as well for python 2.7,
python 3.6 or python 3.7.
Let me know if you want more gory details.
My 2 cents,
Maxime Boissonneault
On 18-02-02 13:32, Baker D.J. wrote:
Hello,
We are evaluating Spack and lmod on our new HPC cluster at the
moment, and a colleague has raised the issue of how best to deal
with python2/3. Python is very popular with our users, and a
typical python environment is getting much more complex especially
with the raise of python3. We would like to be able to use Spack
and lmod to deal with python2/3 and their modules in a nice way.
At the lowest level it would be useful just to be able to
"segregate" python related modules in to separate module
directories -- perhaps one for python 2 and one for python 3. In
other words, so that when users do a module av the python modules
aren't mixed in with with other package modules.
Even better, it would be really useful if users could be protected
from getting python2/3 modules mixed up. We are already starting
to have issues of that sort on our older clusters. I suspect that
one way to deal with this would be to build in conflict statements
in the modules. On the other hand I wondered if it were possible
to deal with python2/3 in the same way as lmod treats compilers.
That is, to design an extension of the current hierarchical system.
I must confess that I'm a bit green -- having only been looking at
Spack and lmod, on and off, for a month or so. Could someone in
the community please advise me? They may have done this sort of
module management with python or another class of modules. I would
be very grateful for any useful advise, please.
Best regards,
David
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!http://sdm.link/slashdot
_______________________________________________
Lmod-users mailing list
lmod-us...@lists.sourceforge.net
<mailto:lmod-us...@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/lmod-users
--
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique
--
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique
