Dear Loris,
On 04/04/2019 11:57, Loris Bennett wrote:
Hi Kenneth,
Kenneth Hoste <[email protected]> writes:
Dear Loris,
On 02/04/2019 17:04, Loris Bennett wrote:
Dear Kenneth,
OK, thanks for clarifying that point.
One thing that surprised me in the context of trying to create a
extension-only easyconfig was that, if I forget to include the primary
program, such as Python 2.7.15, as a dependency, then the primary
program itself, e.g. Python, will be installed as well as the
extensions. This only made itself apparent to me when I loaded the
extensions module and found that the primary program module was then,
due to the conflict, automatically unloaded.
EasyBuild is not doing that automatically trying to be smart of something,
you're somehow telling it to do that...
Or is this only the case
if I also (!) mistakenly have
source_urls = ['http://www.python.org/ftp/%(namelower)s/%(version)s/']
sources = [SOURCE_TGZ]
That's likely the reason, but then you should still be using 'Python' as a
software name, otherwise you shouldn't see the behavior you're describing.
Easyconfigs using the 'Bundle' easyblock should not specify any top-level
'sources'.
We even prevent this in the central easyconfigs repository (the tests will trip
over it), but it's not hard blocked in EasyBuild itself, so if you want to shoot
yourself in the foot, you can. ;)
OK, I have removed the 'sources', but am still now hitting a different
foot.
The Python packages seem to be installed OK, but the sanity check fails
for two modules. However, if I load the corresponding Python version
and set PYTHONPATH by hand, I can import the modules which cause the
failure of the sanity check:
$ module add Python/2.7.15-foss-2018b
$ export
PYTHONPATH=/trinity/shared/easybuild/software/Python/2.7.15-foss-2018b-zedat_extras/lib/python2.7/site-packages/
$ python
Python 2.7.15 (default, Jan 18 2019, 10:12:12)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> import nonconformist
In the log from the build I have
== 2019-04-04 10:43:44,756 build_log.py:162 ERROR EasyBuild crashed with an
error (at
easybuild/software/EasyBuild/3.8.1/lib/python2.7/site-packages/vsc_base-2.8.3-py2.7.egg/vsc/utils/exceptions.py:124
in __init__): Sanity check failed: extensions sanity check failed for 2
extensions: dill, nonconformist
failing sanity check for 'dill' extension: command
"/trinity/shared/easybuild/software/Python/2.7.15-foss-2018b/bin/python -c "import
dill"" failed; output:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named dill,
failing sanity check for 'nonconformist' extension: command
"/trinity/shared/easybuild/software/Python/2.7.15-foss-2018b/bin/python -c "import
nonconformist"" failed; output:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named nonconformist, (at
easybuild/software/EasyBuild/3.8.1/lib/python2.7/site-packages/easybuild_framework-3.8.1-py2.7.egg/easybuild/framework/easyblock.py:2369
in _sanity_check_step)
== 2019-04-04 10:43:44,756 easyblock.py:2870 WARNING build failed (first 300
chars): Sanity check failed: extensions sanity check failed for 2 extensions:
dill, nonconformist
failing sanity check for 'dill' extension: command
"/trinity/shared/easybuild/software/Python/2.7.15-foss-2018b/bin/python -c "import
dill"" failed; output:
Traceback (most recent call last):
File "<string>",
The package directory looks like this:
$ ll
/trinity/shared/easybuild/software/Python/2.7.15-foss-2018b-zedat_extras/lib/python2.7/site-packages/
total 1024
drwxr-xr-x 4 build staff 46 Apr 4 10:15 dill-0.2.9-py2.7.egg
-rw-r--r-- 1 build staff 592 Apr 4 10:43 easy-install.pth
drwxr-xr-x 2 build staff 291 Apr 4 10:16 nonconformist
-rw-r--r-- 1 build staff 829 Apr 4 10:16
nonconformist-2.1.0-py2.7.egg-info
drwxr-xr-x 4 build staff 47 Apr 4 10:19
numpy-1.14.5-py2.7-linux-x86_64.egg
drwxr-xr-x 4 build staff 48 Apr 4 10:43
pandas-0.24.2-py2.7-linux-x86_64.egg
-rw-r--r-- 1 build staff 8246 Apr 4 10:35
scikit_learn-0.19.1-py2.7.egg-info
drwxr-xr-x 4 build staff 47 Apr 4 10:32
scipy-1.1.0-py2.7-linux-x86_64.egg
-rw-r--r-- 1 build staff 1011322 Apr 4 10:16 setuptools-39.1.0-py2.7.egg
-rw-r--r-- 1 build staff 30 Apr 4 10:16 setuptools.pth
-rw-r--r-- 1 build staff 2307 Apr 4 10:15 site.py
-rw-r--r-- 1 build staff 1841 Apr 4 10:15 site.pyc
drwxr-xr-x 27 build staff 4096 Apr 4 10:35 sklearn
Any ideas what I am doing wrong this time?
If you're not using PythonBundle to install this bundle of Python
packages (you should be if you're not), then you have to ensure that the
generated module file updates $PYTHONPATH...
You can do so using:
modextrapaths = {'PYTHONPATH': 'lib/python%(pyshortver)s/site-packages'}
in the easyconfig file (or by using PythonBundle which does it for you).
We need better documentation on writing easyconfig files for (bundles
of) Python packages, I hope to get to that soon...
regards,
Kenneth
Cheers,
Loris
