Hi Maxime,

Thank you for your detailed instructions. We have been experimenting today and 
your method does the job very well. We build mpi4py with python2 & 3. Initially 
we did..

module load python/2..
module load mpi4py/3..
Ran a simple hello world test.

We then loaded the python/3.. module in the same session, repeated the mpi4py 
test and it worked perfectly. Brilliant!

Best regards,
David

From: Maxime Boissonneault [mailto:maxime.boissonnea...@calculquebec.ca]
Sent: Thursday, February 15, 2018 5:02 PM
To: Baker D.J. <d.j.ba...@soton.ac.uk>; lmod-users 
<lmod-us...@lists.sourceforge.net>; easybuild@lists.ugent.be
Subject: Re: [Lmod-users] lmod module sectioning & hierarchies -- dealing with 
python modules

Hi Baker,
Of course. Reposting to lmod and easybuild lists, since I suspect this may be 
interesting to both lists

The first thing. All of our python modules add the same PYTHONPATH:
[mboisson@build-node ~]$ module show python/2.7.14 | grep PYTHONPATH
prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages")
[mboisson@build-node ~]$ module show python/3.5 | grep PYTHONPATH
prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages")
[mboisson@build-node ~]$ module show python/3.6 | grep PYTHONPATH
prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages")

In this PYTHONPATH, there is a single file : sitecustomize.py
This is a file that is always read by any python invokation (see the official 
documentation https://docs.python.org/2/library/site.html  
https://docs.python.org/3/library/site.html )
This file contains various things in our case, but the important part is the 
following:

import sys
import os
import site

# use prefixes from EBPYTHONPREFIXES, so they have lower priority than
# virtualenv-installed packages

if "EBPYTHONPREFIXES" in os.environ:
    postfix = os.path.join('lib', 'python'+sys.version[:3], 'site-packages')
    for prefix in os.environ["EBPYTHONPREFIXES"].split(os.pathsep):
        sitedir = os.path.join(prefix, postfix)
        if os.path.isdir(sitedir):
            site.addsitedir(sitedir)


As you can see from the logic, this means that, depending on the version of 
python which is called, it will add a different path to the site directories 
(through site.addsitedir) in which python will search for modules.

This is then used to make python bindings work automagically for software which 
do have such bindings. For example:
[mboisson@build-node ~]$ module show geos | grep "EBPYTHONPREFIXES\|python"
prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1")

[mboisson@build-node ~]$ ls -lhd 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1/lib/python*
drwxr-sr-x 3 ebuser ebuser 4,0K  4 déc 14:33 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1/lib/python2.7

[mboisson@build-node ~]$ module show igraph | grep "EBPYTHONPREFIXES\|python"
prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1")

[mboisson@build-node ~]$ ls -lhd 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python*
drwxr-sr-x 3 ebuser ebuser 4,0K 26 sep 03:35 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python2.7
drwxr-sr-x 3 ebuser ebuser 4,0K 26 sep 03:35 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python3.5

In these examples, the python bindings for geos will work if you use python2.7, 
and the igraph bindings will work if you use either python2.7 or python3.5. 
Note that we do not add any dependency of these modules over any version of 
python. If a user loads python, the bindings will work, if they don't load 
python, they don't need them to work.

For the few "pure python" modules that we offer (scipy-stack, which is defined 
here https://www.scipy.org/stackspec.html  and mpi4py because different 
versions need to be provided depending on the MPI version that is loaded), we 
also use this to make the same module work with any version of python:

[mboisson@build-node ~]$ module show scipy-stack | grep 
"EBPYTHONPREFIXES\|python\""
prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b")
depends_on("python")

[mboisson@build-node ~]$ ls -lhd 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python*
drwxr-sr-x 3 ebuser ebuser 4,0K  4 déc 22:33 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python2.7
drwxr-sr-x 3 ebuser ebuser 4,0K  4 déc 22:33 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python3.5
drwxr-sr-x 3 ebuser ebuser 4,0K  4 déc 22:34 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python3.6

[mboisson@build-node ~]$ module show mpi4py | grep "EBPYTHONPREFIXES\|python\""
prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0")
depends_on("python")

[mboisson@build-node ~]$ ls -lhd 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python*
drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:31 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python2.7
drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:33 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python3.5
drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:34 
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python3.6


In this case, since those modules are only useful for python, we do add a 
"depends_on("python")" marker, but we do not specify any version. If a user 
already had python loaded, it will work with that one. If they had not, it will 
load our default version.

I hope this is useful.

Maxime


On 18-02-15 11:36, Baker D.J. wrote:
Hi Maxime,

Thank you again for your email, sent via the lmod mailing list, regarding 
dealing with python modules. It would be really useful if you could please 
elaborate on your python setup. I've got the basic idea that the 
sitecustomize.py file is generating a new "search path" depending upon the 
version of python being used - the prefix relating to the packages and the 
postfix relating to the python version. Do you have a concrete example that you 
could share please?

Furthermore, I understand that the scipy-stack is a software stack providing 
python2/3 and a host of modules (numpy, scripy, etc) and there would be a 
module for this stack. Given that, how do I select whether I need to use python 
2 or 3 in this setup?

Your clarification would be really appreciated, please.

Best regards,
David

From: Maxime Boissonneault [mailto:maxime.boissonnea...@calculquebec.ca]
Sent: Friday, February 02, 2018 9:14 PM
To: Baker D.J. <d.j.ba...@soton.ac.uk><mailto:d.j.ba...@soton.ac.uk>; 
lmod-us...@lists.sourceforge.net<mailto:lmod-us...@lists.sourceforge.net>
Subject: Re: [Lmod-users] lmod module sectioning & hierarchies -- dealing with 
python modules

Hi David,
My advice regarding python modules : don't. Python already has very nice 
handling of dependencies, don't try to reproduce them through modules. Instead, 
educate your users to use pip and virtualenv to install whatever they need with 
their dependencies.

Most python packages these days have a binary version that is very 
straightforward to install. If you prefer to build them for performance reason, 
create a python "wheel house" instead. https://wheel.readthedocs.io/en/stable/
A wheel house is simply a directory on your cluster in which you hosts binary 
wheels that you will have compiled for your users.

Using $PIP_CONFIG_FILE, you can configure pip so that it first looks into your 
wheel house first, and looks online only if what it needs isn't found in the 
wheel house.

That is what we do on our cluster, and it works very very well.

The only python modules that we have are bundles (scipy-stack, mpi4py). For 
those also, we do not use PYTHONPATH. PYTHONPATH is very intrusive, and only 
works for one version of python or the other, even though python is designed to 
be able to handle both python2 and python3 at the same time since packages get 
installed in pythonX.Y/site-packages/. Instead of using PYTHONPATH, we define a 
difference environment variable  (EBPYTHONPREFIXES), and we have a 
sitecustomize.py   file which looks at this directory, and will add paths 
internally based on the version of python that is used :
        postfix = os.path.join('lib', 'python'+sys.version[:3], 'site-packages')
        nixstoresitepackages = os.path.join(sys.prefix, postfix)
        if nixstoresitepackages in sys.path:
            site.addsitedir(os.path.join(newprefix, postfix))

That way, our "scipy-stack" module works just as well for python 2.7, python 
3.6 or python 3.7.

Let me know if you want more gory details.


My 2 cents,

Maxime Boissonneault

On 18-02-02 13:32, Baker D.J. wrote:
Hello,

We are evaluating Spack and lmod on our new HPC cluster at the moment, and a 
colleague has raised the issue of how best to deal with python2/3. Python is 
very popular with our users, and a typical python environment is getting much 
more complex especially with the raise of python3. We would like to be able to 
use Spack and lmod to deal with python2/3 and their modules in a nice way.

At the lowest level it would be useful just to be able to "segregate" python 
related modules in to separate module directories -- perhaps one for python 2 
and one for python 3. In other words, so that when users do a module av the 
python modules aren't mixed in with with other package modules.

Even better, it would be really useful if users could be protected from getting 
python2/3 modules mixed up. We are already starting to have issues of that sort 
on our older clusters. I suspect that one way to deal with this would be to 
build in conflict statements in the modules. On the other hand I wondered if it 
were possible to deal with python2/3 in the same way as lmod treats compilers. 
That is, to design an extension of the current hierarchical system.

I must confess that I'm a bit green -- having only been looking at Spack and 
lmod, on and off, for a month or so. Could someone in the community please 
advise me? They may have done this sort of module management with python or 
another class of modules. I would be very grateful for any useful advise, 
please.

Best regards,
David





------------------------------------------------------------------------------

Check out the vibrant tech community on one of the world's most

engaging tech sites, Slashdot.org! http://sdm.link/slashdot





_______________________________________________

Lmod-users mailing list

lmod-us...@lists.sourceforge.net<mailto:lmod-us...@lists.sourceforge.net>

https://lists.sourceforge.net/lists/listinfo/lmod-users



--

---------------------------------

Maxime Boissonneault

Analyste de calcul - Calcul Québec, Université Laval

Président - Comité de coordination du soutien à la recherche de Calcul Québec

Team lead - Research Support National Team, Compute Canada

Instructeur Software Carpentry

Ph. D. en physique



--

---------------------------------

Maxime Boissonneault

Analyste de calcul - Calcul Québec, Université Laval

Président - Comité de coordination du soutien à la recherche de Calcul Québec

Team lead - Research Support National Team, Compute Canada

Instructeur Software Carpentry

Ph. D. en physique

Reply via email to