Hi Maxime, Thank you for your detailed instructions. We have been experimenting today and your method does the job very well. We build mpi4py with python2 & 3. Initially we did..
module load python/2.. module load mpi4py/3.. Ran a simple hello world test. We then loaded the python/3.. module in the same session, repeated the mpi4py test and it worked perfectly. Brilliant! Best regards, David From: Maxime Boissonneault [mailto:maxime.boissonnea...@calculquebec.ca] Sent: Thursday, February 15, 2018 5:02 PM To: Baker D.J. <d.j.ba...@soton.ac.uk>; lmod-users <lmod-us...@lists.sourceforge.net>; firstname.lastname@example.org Subject: Re: [Lmod-users] lmod module sectioning & hierarchies -- dealing with python modules Hi Baker, Of course. Reposting to lmod and easybuild lists, since I suspect this may be interesting to both lists The first thing. All of our python modules add the same PYTHONPATH: [mboisson@build-node ~]$ module show python/2.7.14 | grep PYTHONPATH prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages") [mboisson@build-node ~]$ module show python/3.5 | grep PYTHONPATH prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages") [mboisson@build-node ~]$ module show python/3.6 | grep PYTHONPATH prepend_path("PYTHONPATH","/cvmfs/soft.computecanada.ca/easybuild/python/site-packages") In this PYTHONPATH, there is a single file : sitecustomize.py This is a file that is always read by any python invokation (see the official documentation https://docs.python.org/2/library/site.html https://docs.python.org/3/library/site.html ) This file contains various things in our case, but the important part is the following: import sys import os import site # use prefixes from EBPYTHONPREFIXES, so they have lower priority than # virtualenv-installed packages if "EBPYTHONPREFIXES" in os.environ: postfix = os.path.join('lib', 'python'+sys.version[:3], 'site-packages') for prefix in os.environ["EBPYTHONPREFIXES"].split(os.pathsep): sitedir = os.path.join(prefix, postfix) if os.path.isdir(sitedir): site.addsitedir(sitedir) As you can see from the logic, this means that, depending on the version of python which is called, it will add a different path to the site directories (through site.addsitedir) in which python will search for modules. This is then used to make python bindings work automagically for software which do have such bindings. For example: [mboisson@build-node ~]$ module show geos | grep "EBPYTHONPREFIXES\|python" prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1") [mboisson@build-node ~]$ ls -lhd /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1/lib/python* drwxr-sr-x 3 ebuser ebuser 4,0K 4 déc 14:33 /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/geos/3.6.1/lib/python2.7 [mboisson@build-node ~]$ module show igraph | grep "EBPYTHONPREFIXES\|python" prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1") [mboisson@build-node ~]$ ls -lhd /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python* drwxr-sr-x 3 ebuser ebuser 4,0K 26 sep 03:35 /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python2.7 drwxr-sr-x 3 ebuser ebuser 4,0K 26 sep 03:35 /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/igraph/0.7.1/lib/python3.5 In these examples, the python bindings for geos will work if you use python2.7, and the igraph bindings will work if you use either python2.7 or python3.5. Note that we do not add any dependency of these modules over any version of python. If a user loads python, the bindings will work, if they don't load python, they don't need them to work. For the few "pure python" modules that we offer (scipy-stack, which is defined here https://www.scipy.org/stackspec.html and mpi4py because different versions need to be provided depending on the MPI version that is loaded), we also use this to make the same module work with any version of python: [mboisson@build-node ~]$ module show scipy-stack | grep "EBPYTHONPREFIXES\|python\"" prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b") depends_on("python") [mboisson@build-node ~]$ ls -lhd /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python* drwxr-sr-x 3 ebuser ebuser 4,0K 4 déc 22:33 /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python2.7 drwxr-sr-x 3 ebuser ebuser 4,0K 4 déc 22:33 /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python3.5 drwxr-sr-x 3 ebuser ebuser 4,0K 4 déc 22:34 /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/scipy-stack/2017b/lib/python3.6 [mboisson@build-node ~]$ module show mpi4py | grep "EBPYTHONPREFIXES\|python\"" prepend_path("EBPYTHONPREFIXES","/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0") depends_on("python") [mboisson@build-node ~]$ ls -lhd /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python* drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:31 /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python2.7 drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:33 /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python3.5 drwxr-sr-x 3 ebuser ebuser 4,0K 14 déc 13:34 /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/mpi4py/3.0.0/lib/python3.6 In this case, since those modules are only useful for python, we do add a "depends_on("python")" marker, but we do not specify any version. If a user already had python loaded, it will work with that one. If they had not, it will load our default version. I hope this is useful. Maxime On 18-02-15 11:36, Baker D.J. wrote: Hi Maxime, Thank you again for your email, sent via the lmod mailing list, regarding dealing with python modules. It would be really useful if you could please elaborate on your python setup. I've got the basic idea that the sitecustomize.py file is generating a new "search path" depending upon the version of python being used - the prefix relating to the packages and the postfix relating to the python version. Do you have a concrete example that you could share please? Furthermore, I understand that the scipy-stack is a software stack providing python2/3 and a host of modules (numpy, scripy, etc) and there would be a module for this stack. Given that, how do I select whether I need to use python 2 or 3 in this setup? Your clarification would be really appreciated, please. Best regards, David From: Maxime Boissonneault [mailto:maxime.boissonnea...@calculquebec.ca] Sent: Friday, February 02, 2018 9:14 PM To: Baker D.J. <d.j.ba...@soton.ac.uk><mailto:d.j.ba...@soton.ac.uk>; lmod-us...@lists.sourceforge.net<mailto:lmod-us...@lists.sourceforge.net> Subject: Re: [Lmod-users] lmod module sectioning & hierarchies -- dealing with python modules Hi David, My advice regarding python modules : don't. Python already has very nice handling of dependencies, don't try to reproduce them through modules. Instead, educate your users to use pip and virtualenv to install whatever they need with their dependencies. Most python packages these days have a binary version that is very straightforward to install. If you prefer to build them for performance reason, create a python "wheel house" instead. https://wheel.readthedocs.io/en/stable/ A wheel house is simply a directory on your cluster in which you hosts binary wheels that you will have compiled for your users. Using $PIP_CONFIG_FILE, you can configure pip so that it first looks into your wheel house first, and looks online only if what it needs isn't found in the wheel house. That is what we do on our cluster, and it works very very well. The only python modules that we have are bundles (scipy-stack, mpi4py). For those also, we do not use PYTHONPATH. PYTHONPATH is very intrusive, and only works for one version of python or the other, even though python is designed to be able to handle both python2 and python3 at the same time since packages get installed in pythonX.Y/site-packages/. Instead of using PYTHONPATH, we define a difference environment variable (EBPYTHONPREFIXES), and we have a sitecustomize.py file which looks at this directory, and will add paths internally based on the version of python that is used : postfix = os.path.join('lib', 'python'+sys.version[:3], 'site-packages') nixstoresitepackages = os.path.join(sys.prefix, postfix) if nixstoresitepackages in sys.path: site.addsitedir(os.path.join(newprefix, postfix)) That way, our "scipy-stack" module works just as well for python 2.7, python 3.6 or python 3.7. Let me know if you want more gory details. My 2 cents, Maxime Boissonneault On 18-02-02 13:32, Baker D.J. wrote: Hello, We are evaluating Spack and lmod on our new HPC cluster at the moment, and a colleague has raised the issue of how best to deal with python2/3. Python is very popular with our users, and a typical python environment is getting much more complex especially with the raise of python3. We would like to be able to use Spack and lmod to deal with python2/3 and their modules in a nice way. At the lowest level it would be useful just to be able to "segregate" python related modules in to separate module directories -- perhaps one for python 2 and one for python 3. In other words, so that when users do a module av the python modules aren't mixed in with with other package modules. Even better, it would be really useful if users could be protected from getting python2/3 modules mixed up. We are already starting to have issues of that sort on our older clusters. I suspect that one way to deal with this would be to build in conflict statements in the modules. On the other hand I wondered if it were possible to deal with python2/3 in the same way as lmod treats compilers. That is, to design an extension of the current hierarchical system. I must confess that I'm a bit green -- having only been looking at Spack and lmod, on and off, for a month or so. Could someone in the community please advise me? They may have done this sort of module management with python or another class of modules. I would be very grateful for any useful advise, please. Best regards, David ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Lmod-users mailing list lmod-us...@lists.sourceforge.net<mailto:lmod-us...@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/lmod-users -- --------------------------------- Maxime Boissonneault Analyste de calcul - Calcul Québec, Université Laval Président - Comité de coordination du soutien à la recherche de Calcul Québec Team lead - Research Support National Team, Compute Canada Instructeur Software Carpentry Ph. D. en physique -- --------------------------------- Maxime Boissonneault Analyste de calcul - Calcul Québec, Université Laval Président - Comité de coordination du soutien à la recherche de Calcul Québec Team lead - Research Support National Team, Compute Canada Instructeur Software Carpentry Ph. D. en physique