On 16-08-22 09:50, Kenneth Hoste wrote:
For Python packages, R packages, Perl packages, Lua packages, etc.,
we usually instruct our users to either install them in their home
(using pip --user for Python for example), or using a virtual
environment (for Python).
In order to avoid build headaches for our users with this strategy,
we use python wheels (http://pythonwheels.com/) to pre-build most
used Python packages. This way, installing the packages (either in
their home or a virtual environment) takes mere seconds.
I guess this is a matter of site policies, but we try to avoid letting
users handle this themselves, for a variety of reasons (one being that
our setup involves a single (set of) login nodes for multiple clusters
with different operating systems).
We also do have multiple login nodes (and multiple clusters) with a
shared filesystem. We use different wheelhouses to handle the
installation of the correct wheel based on the cluster they install the
wheels from. If they create their virtual environment within their job,
in ramdisk, it also has the advantage of decoupling their job from the
filesystem, and render their job self-contained. Given that we do our
job correctly, they are also guaranteed to install the correct wheel
based on the cluster they are running on.
It would be nice to have wheel support for Python packages within
EasyBuild.
It is already supported, see for example
https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/t/testpath/testpath-0.3-intel-2016a-Python-2.7.11.eb
.
Thanks, but what I meant is to have EasyBuild generate a wheel house,
rather than install them from a wheel (i.e. build the wheels rather
than install them as modules).
Maxime
