On 16-08-22 09:50, Kenneth Hoste wrote:
For Python packages, R packages, Perl packages, Lua packages, etc., we usually instruct our users to either install them in their home (using pip --user for Python for example), or using a virtual environment (for Python). In order to avoid build headaches for our users with this strategy, we use python wheels (http://pythonwheels.com/) to pre-build most used Python packages. This way, installing the packages (either in their home or a virtual environment) takes mere seconds.

I guess this is a matter of site policies, but we try to avoid letting users handle this themselves, for a variety of reasons (one being that our setup involves a single (set of) login nodes for multiple clusters with different operating systems).

We also do have multiple login nodes (and multiple clusters) with a shared filesystem. We use different wheelhouses to handle the installation of the correct wheel based on the cluster they install the wheels from. If they create their virtual environment within their job, in ramdisk, it also has the advantage of decoupling their job from the filesystem, and render their job self-contained. Given that we do our job correctly, they are also guaranteed to install the correct wheel based on the cluster they are running on.

It would be nice to have wheel support for Python packages within EasyBuild.

It is already supported, see for example https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/t/testpath/testpath-0.3-intel-2016a-Python-2.7.11.eb .

Thanks, but what I meant is to have EasyBuild generate a wheel house, rather than install them from a wheel (i.e. build the wheels rather than install them as modules).


Reply via email to