Hi Bernd,
On 03/06/16 19:13, Bernd Mohr wrote:
Kenneth,
might be my weird setup (which was created by collecting all kind of
junk over the past 20 years in my dot files on various systems ;-)
In my dot files:
1) I do a "module load lmod" first
(which prepends /opt/local/lmod/lmod/libexec)
2) Then I prepend /opt/local/bin to my path.
In my installation notes for EB I have
setenv PATH /opt/local/lmod/lmod/libexec:${PATH}
(Without this (at least at some point in time) EB did not bootstrap
syaing in cannot find lmod)
So that could explain it, if
if bootstrap does a module purge or modle unload lmod,
it actually removes BOTH instances of /opt/local/lmod/lmod/libexec
from the PATh leaving /opt/local/bin ;-)
I remember before discussing this behavior with Markus, and we noticed
that different module installations behave different here.
Thanks for clarifying that.
EasyBuild or the bootstrap script does not do a 'module purge' (anymore,
it used to though) or a 'module unload lmod' directly,
but it does reset $MODULEPATH...
That causes Lmod to mark the 'lmod' module as inactive, which leads to
removing /opt/local/lmod/lmod/libexec from $PATH...
I also checked with Robert: running the 'lmod' from a symlinked location
basically doesn't work (and it never has), because Lmod looks for things
relative to the location of the binary (without resolving a possible
symlink, I guess that would solve it).
There's a couple of things we can/should do to improve the current
situation:
* make EasyBuild check for a hard failing Lmod (e.g. check for empty
stdout + 'stack traceback' in stderr)
* make sure that the 'lmod' being used by EasyBuild is *not* a symlink
* run some kind of sanity check every time we select an 'lmod' to use,
to make sure it actually works (and is what we expect it to be)
Any of these would have made at least the actual problem a lot more easy
to spot... :-)
regards,
Kenneth
Bernd
On Fri, Jun 03, 2016 at 06:51:20PM +0200, Kenneth Hoste wrote:
Hi Bernd,
On 03/06/16 18:32, Bernd Mohr wrote:
Kenneth,
I had a symlink /opt/local/bin/lmod --> /opt/local/lmod/lmod/libexec/lmod
(in the same way as for lua).
After removing the symlink, the installation works now.
What is weird, is that this worked before: I just checked my other SuSE
installation with EB 2.0, 2.1 and 2.2 and there it was no problem.
I only had to make sure
that /opt/local/lmod/lmod/libexec was in $PATH before /opt/local/bin
Glad to hear we have it figured out, but I'm still puzzled...
i) Why does Lmod crash like that when it's being called via a symlink?
That must be some kind of regression with a recent Lmod version,
which may explain why you weren't seeing it before.
ii) Why does EasyBuild suddenly pick up 'lmod' via $PATH instead of
using $LMOD_CMD like it did before?
I checked the code, and the checking for 'lmod' is indeed done twice.
That is, the ModulesTool instance is creating twice
Once via main, where it is being used to check whether modules are
available, and then once for every software installation being
performed.
For some reason unclear to me, the 1st time the 'lmod' found via
$PATH is the one in /opt/local/lmod/lmod/libexec, which is the same
as the path indicated by $LMOD_CMD, and things work fine.
The 2nd time however, it finds the 'lmod' in /opt/local/bin via
$PATH, making it ignore the path in $LMOD_CMD.
(the latter is mostly for backward compatibility, picking up
$LMOD_CMD if it's there and ignoring $PATH would be better, we can
change that in EasyBuild v3.0).
It seems that a 'module use' command being run throws out
/opt/local/lmod/lmod/libexec from $PATH for some reason (or, doesn't
include it again)...
How are you included /opt/local/lmod/lmod/libexec in $PATH exactly?
Any ideas why it would no longer be there after a 'module use'?
Anyway, seems like the real bug is in Lmod, I'll pick this up with
Robert. ;-)
regards,
Kenneth
Anyhow
Thanks!
Bernd
On Fri, Jun 03, 2016 at 06:21:06PM +0200, Kenneth Hoste wrote:
Hi Bernd,
Seems like there's a problem with your Lmod setup/installation:
== 2016-06-03 17:55:47,165 modules.py:613 DEBUG Output of module
command '/opt/local/bin/lmod python load EasyBuild/2.8.1': stdout: ;
stderr: /opt/local/lua/lua/bin/lua: /opt/local/bin/lmod:84: module
'myGlobals' not found:
no field package.preload['myGlobals']
no file '/opt/local/bin/myGlobals.lua'
no file '/opt/local/bin/../tools/myGlobals.lua'
no file '/opt/local/bin/../shells/myGlobals.lua'
no file '/opt/local/bin/myGlobals/init.lua'
no file './myGlobals.lua'
no file '/opt/local/lua/5.1.4.8/share/lua/5.1/myGlobals.lua'
no file '/opt/local/lua/5.1.4.8/share/lua/5.1/myGlobals/init.lua'
no file '/opt/local/lua/5.1.4.8/lib/lua/5.1/myGlobals.lua'
no file '/opt/local/lua/5.1.4.8/lib/lua/5.1/myGlobals/init.lua'
no file '/opt/local/bin/../lib/myGlobals.so'
no file './myGlobals.so'
no file '/opt/local/lua/5.1.4.8/lib/lua/5.1/myGlobals.so'
no file '/opt/local/lua/5.1.4.8/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'require'
/opt/local/bin/lmod:84: in main chunk
[C]: ?
The $PYTHONPATH setting I was grabbing from the log earlier isn't
the value that is active when the 'import easybuild' is being run.
The problem with the load is not detected by EasyBuild because Lmod
is exiting normally; usually it includes a 'false' in the stdout to
make the evaluation of the environment changes it spits out fail.
Some commands are running fine:
"/opt/local/lmod/lmod/libexec/lmod python --version" spits out
"Modules based on Lua: Version 6.4 2016-06-01 17:42"
"/opt/local/lmod/lmod/libexec/lmod python --terse list"
"/opt/local/lmod/lmod/libexec/lmod python --ters avail"
"/opt/local/lmod/lmod/libexec/lmod python show Core/EasyBuild/2.8.1"
=> all work fine (i.e. they don't crash)
But then suddenly "/opt/local/bin/lmod python --terse list" goes
down hard like above.
Note the change in location for 'lmod' itself...
Shortly before, the log is showing this:
== 2016-06-03 17:55:15,958 filetools.py:217 INFO Command lmod found
at /opt/local/bin/lmod
== 2016-06-03 17:55:15,958 filetools.py:217 INFO Command lmod found
at /opt/local/bin/lmod
== 2016-06-03 17:55:15,958 modules.py:179 DEBUG Different paths
found for module command 'lmod' via which/$PATH and $LMOD_CMD: lmod
vs /opt/local/lmod/lmod/libexec/lmod
== 2016-06-03 17:55:15,959 modules.py:185 DEBUG Using command lmod
== 2016-06-03 17:55:15,959 filetools.py:217 INFO Command lmod found
at /opt/local/bin/lmod
== 2016-06-03 17:55:15,959 modules.py:253 INFO Full path for module
command is /opt/local/bin/lmod, so using it
At some point EasyBuild switches from
/opt/local/lmod/lmod/libexec/lmod ($LMOD_CMD) to opt/local/bin/lmod
(via $PATH).
That's very stupid/wrong, I'll check on that, but it seems to
explain what's going on...
regards,
Kenneth
On 03/06/16 18:00, Bernd Mohr wrote:
debug log attached.
Executing after the failed installation:
% setenv PYTHONPATH
/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages
% python -c "import easybuild"
Also
% setenv PYTHONPATH
/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages:/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib64/python2.7/site-packages
% python -c "import easybuild"
no error / output
Bernd
On Fri, Jun 03, 2016 at 05:47:57PM +0200, Kenneth Hoste wrote:
On 03/06/16 17:41, Bernd Mohr wrote:
OK, so everything *is* there, but for some reason Python isn't
picking up on it. Strange...
You do have this, right?
ls -l
/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/easybuild/__init__.py
Yes
Sorry, I meant:
cat
/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages/easy-install.pth
import sys; sys.__plen = len(sys.path)
./distribute-0.7.3-py2.7.egg
./setuptools-22.0.2-py2.7.egg
./vsc_install-0.10.6-py2.7.egg
./vsc_base-2.5.1-py2.7.egg
./easybuild_framework-2.8.1-py2.7.egg
./easybuild_easyblocks-2.8.1-py2.7.egg
./easybuild_easyconfigs-2.8.1-py2.7.egg
import sys; new = sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p =
getattr(sys, '__egginsert', 0); sys.path[p:p] = new; sys.__egginsert = p +
len(new)
Looks fine...
and let's throw in this too:
cat
/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages/setuptools.pth
./setuptools-22.0.2-py2.7.egg
This is fine too.
Any PYTHONPATH issue?
I tried unsetenv PYTHONPATH but doid not change anything
From the log you provided, right before the "/usr/bin/python -c
'import easybuid' " that fails.
== 2016-06-03 16:53:19,262 environment.py:97 INFO Environment
variable PYTHONPATH set to
/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages:/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib64/python2.7/site-packages
(previous value: ...)
So, looks fine. I have no idea why the import fails...
OK, let's dig a little deeper.
Can you rebootstrap with debugging enabled, and send back the full log?
export EASYBUILD_BOOTSTRAP_DEBUG=1
python bootstrap_eb.py /opt/local/easybuild-2.8.1 2>&1 | tee
eb-debug.log
Also, let's try this (after the failing bootstrap):
PYTHONPATH=/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages
python -c "import easybuild"
regards,
Kenneth
Bernd
regards,
Kenneth
;-)
Bernd
regards,
Kenneth
On 03/06/16 17:00, Bernd Mohr wrote:
Fresh install with
% ./bootstrap_eb.py /opt/local/easybuild-2.8.1
on a fresh (empty) SuSE LEAP 42.1 system.
% /usr/bin/python --version
Python 2.7.9
Failure in Stage2:
+++ STAGE 2: installing EasyBuild in /opt/local/easybuild-2.8.1 with EasyBuild
from stage 1...
== temporary log file in case of crash /tmp/eb-ZufCws/easybuild-bC72PS.log
== processing EasyBuild easyconfig /tmp/tmpnlyQGk/EasyBuild-2.8.1.eb
== building and installing Core/EasyBuild/2.8.1...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
== FAILED: Installation ended unsuccessfully (build directory:
/tmp/tmpnlyQGk/EasyBuild/2.8.1/dummy-dummy): build failed (first 300 chars): Sanity check
failed: EasyBuild failed to install, cmd '/usr/bin/python -c "import
easybuild"' (stdin: None) output: Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named easybuild
, sanity check command eb --version exited with code 1 (output: /usr/bi
Any ideas?
Full installation log attached.
Bernd
--
Dr.-Ing. Bernd Mohr
Juelich Supercomputing Centre
Institute for Advanced Simulation
E-Mail: [email protected]
WWW:
http://www.fz-juelich.de/SharedDocs/Personen/IAS/JSC/EN/staff/mohr_b.html
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
