On 22/08/16 15:59, Maxime Boissonneault wrote:
On 16-08-22 09:50, Kenneth Hoste wrote:
For Python packages, R packages, Perl packages, Lua packages, etc.,
we usually instruct our users to either install them in their home
(using pip --user for Python for example), or using a virtual
environment (for Python).
In order to avoid build headaches for our users with this strategy,
we use python wheels (http://pythonwheels.com/) to pre-build most
used Python packages. This way, installing the packages (either in
their home or a virtual environment) takes mere seconds.
I guess this is a matter of site policies, but we try to avoid
letting users handle this themselves, for a variety of reasons (one
being that our setup involves a single (set of) login nodes for
multiple clusters with different operating systems).
We also do have multiple login nodes (and multiple clusters) with a
shared filesystem. We use different wheelhouses to handle the
installation of the correct wheel based on the cluster they install
the wheels from. If they create their virtual environment within their
job, in ramdisk, it also has the advantage of decoupling their job
from the filesystem, and render their job self-contained. Given that
we do our job correctly, they are also guaranteed to install the
correct wheel based on the cluster they are running on.
It would be nice to have wheel support for Python packages within
EasyBuild.
It is already supported, see for example
https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/t/testpath/testpath-0.3-intel-2016a-Python-2.7.11.eb
.
Thanks, but what I meant is to have EasyBuild generate a wheel house,
rather than install them from a wheel (i.e. build the wheels rather
than install them as modules).
Hmm, what would that involve, exactly?
This sounds like it may be out of scope for EasyBuild, although I'm not
sure...
Maybe open an issue with a feature request in the easybuild-framework
repo with more details (or a pull request that adds this functionality ;-))?
regards,
Kenneth
