Kenneth,
might be my weird setup (which was created by collecting all kind of
junk over the past 20 years in my dot files on various systems ;-)
In my dot files:
1) I do a "module load lmod" first
(which prepends /opt/local/lmod/lmod/libexec)
2) Then I prepend /opt/local/bin to my path.
In my installation notes for EB I have
setenv PATH /opt/local/lmod/lmod/libexec:${PATH}
(Without this (at least at some point in time) EB did not bootstrap
syaing in cannot find lmod)
So that could explain it, if
if bootstrap does a module purge or modle unload lmod,
it actually removes BOTH instances of /opt/local/lmod/lmod/libexec
from the PATh leaving /opt/local/bin ;-)
I remember before discussing this behavior with Markus, and we noticed
that different module installations behave different here.
Bernd
On Fri, Jun 03, 2016 at 06:51:20PM +0200, Kenneth Hoste wrote:
> Hi Bernd,
>
> On 03/06/16 18:32, Bernd Mohr wrote:
> >Kenneth,
> >
> >I had a symlink /opt/local/bin/lmod --> /opt/local/lmod/lmod/libexec/lmod
> >(in the same way as for lua).
> >
> >After removing the symlink, the installation works now.
> >
> >What is weird, is that this worked before: I just checked my other SuSE
> >installation with EB 2.0, 2.1 and 2.2 and there it was no problem.
> >
> >I only had to make sure
> >that /opt/local/lmod/lmod/libexec was in $PATH before /opt/local/bin
>
> Glad to hear we have it figured out, but I'm still puzzled...
>
> i) Why does Lmod crash like that when it's being called via a symlink?
>
> That must be some kind of regression with a recent Lmod version,
> which may explain why you weren't seeing it before.
>
>
> ii) Why does EasyBuild suddenly pick up 'lmod' via $PATH instead of
> using $LMOD_CMD like it did before?
>
> I checked the code, and the checking for 'lmod' is indeed done twice.
> That is, the ModulesTool instance is creating twice
>
> Once via main, where it is being used to check whether modules are
> available, and then once for every software installation being
> performed.
>
> For some reason unclear to me, the 1st time the 'lmod' found via
> $PATH is the one in /opt/local/lmod/lmod/libexec, which is the same
> as the path indicated by $LMOD_CMD, and things work fine.
>
> The 2nd time however, it finds the 'lmod' in /opt/local/bin via
> $PATH, making it ignore the path in $LMOD_CMD.
> (the latter is mostly for backward compatibility, picking up
> $LMOD_CMD if it's there and ignoring $PATH would be better, we can
> change that in EasyBuild v3.0).
>
> It seems that a 'module use' command being run throws out
> /opt/local/lmod/lmod/libexec from $PATH for some reason (or, doesn't
> include it again)...
>
> How are you included /opt/local/lmod/lmod/libexec in $PATH exactly?
> Any ideas why it would no longer be there after a 'module use'?
>
> Anyway, seems like the real bug is in Lmod, I'll pick this up with
> Robert. ;-)
>
>
> regards,
>
> Kenneth
>
> >
> >Anyhow
> >Thanks!
> >Bernd
> >
> >On Fri, Jun 03, 2016 at 06:21:06PM +0200, Kenneth Hoste wrote:
> >>Hi Bernd,
> >>
> >>Seems like there's a problem with your Lmod setup/installation:
> >>
> >>== 2016-06-03 17:55:47,165 modules.py:613 DEBUG Output of module
> >>command '/opt/local/bin/lmod python load EasyBuild/2.8.1': stdout: ;
> >>stderr: /opt/local/lua/lua/bin/lua: /opt/local/bin/lmod:84: module
> >>'myGlobals' not found:
> >> no field package.preload['myGlobals']
> >> no file '/opt/local/bin/myGlobals.lua'
> >> no file '/opt/local/bin/../tools/myGlobals.lua'
> >> no file '/opt/local/bin/../shells/myGlobals.lua'
> >> no file '/opt/local/bin/myGlobals/init.lua'
> >> no file './myGlobals.lua'
> >> no file '/opt/local/lua/5.1.4.8/share/lua/5.1/myGlobals.lua'
> >> no file '/opt/local/lua/5.1.4.8/share/lua/5.1/myGlobals/init.lua'
> >> no file '/opt/local/lua/5.1.4.8/lib/lua/5.1/myGlobals.lua'
> >> no file '/opt/local/lua/5.1.4.8/lib/lua/5.1/myGlobals/init.lua'
> >> no file '/opt/local/bin/../lib/myGlobals.so'
> >> no file './myGlobals.so'
> >> no file '/opt/local/lua/5.1.4.8/lib/lua/5.1/myGlobals.so'
> >> no file '/opt/local/lua/5.1.4.8/lib/lua/5.1/loadall.so'
> >>stack traceback:
> >> [C]: in function 'require'
> >> /opt/local/bin/lmod:84: in main chunk
> >> [C]: ?
> >>
> >>
> >>The $PYTHONPATH setting I was grabbing from the log earlier isn't
> >>the value that is active when the 'import easybuild' is being run.
> >>
> >>The problem with the load is not detected by EasyBuild because Lmod
> >>is exiting normally; usually it includes a 'false' in the stdout to
> >>make the evaluation of the environment changes it spits out fail.
> >>
> >>Some commands are running fine:
> >>
> >> "/opt/local/lmod/lmod/libexec/lmod python --version" spits out
> >>"Modules based on Lua: Version 6.4 2016-06-01 17:42"
> >>
> >> "/opt/local/lmod/lmod/libexec/lmod python --terse list"
> >> "/opt/local/lmod/lmod/libexec/lmod python --ters avail"
> >> "/opt/local/lmod/lmod/libexec/lmod python show Core/EasyBuild/2.8.1"
> >> => all work fine (i.e. they don't crash)
> >>
> >>But then suddenly "/opt/local/bin/lmod python --terse list" goes
> >>down hard like above.
> >>
> >>Note the change in location for 'lmod' itself...
> >>
> >>Shortly before, the log is showing this:
> >>
> >>== 2016-06-03 17:55:15,958 filetools.py:217 INFO Command lmod found
> >>at /opt/local/bin/lmod
> >>== 2016-06-03 17:55:15,958 filetools.py:217 INFO Command lmod found
> >>at /opt/local/bin/lmod
> >>== 2016-06-03 17:55:15,958 modules.py:179 DEBUG Different paths
> >>found for module command 'lmod' via which/$PATH and $LMOD_CMD: lmod
> >>vs /opt/local/lmod/lmod/libexec/lmod
> >>== 2016-06-03 17:55:15,959 modules.py:185 DEBUG Using command lmod
> >>== 2016-06-03 17:55:15,959 filetools.py:217 INFO Command lmod found
> >>at /opt/local/bin/lmod
> >>== 2016-06-03 17:55:15,959 modules.py:253 INFO Full path for module
> >>command is /opt/local/bin/lmod, so using it
> >>
> >>
> >>At some point EasyBuild switches from
> >>/opt/local/lmod/lmod/libexec/lmod ($LMOD_CMD) to opt/local/bin/lmod
> >>(via $PATH).
> >>
> >>That's very stupid/wrong, I'll check on that, but it seems to
> >>explain what's going on...
> >>
> >>
> >>
> >>regards,
> >>
> >>Kenneth
> >>
> >>On 03/06/16 18:00, Bernd Mohr wrote:
> >>>debug log attached.
> >>>
> >>>Executing after the failed installation:
> >>>
> >>>% setenv PYTHONPATH
> >>>/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages
> >>>% python -c "import easybuild"
> >>>
> >>>Also
> >>>
> >>>% setenv PYTHONPATH
> >>>/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages:/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib64/python2.7/site-packages
> >>>% python -c "import easybuild"
> >>>
> >>>no error / output
> >>>
> >>>Bernd
> >>>
> >>>
> >>>On Fri, Jun 03, 2016 at 05:47:57PM +0200, Kenneth Hoste wrote:
> >>>>On 03/06/16 17:41, Bernd Mohr wrote:
> >>>>>>OK, so everything *is* there, but for some reason Python isn't
> >>>>>>picking up on it. Strange...
> >>>>>>
> >>>>>>You do have this, right?
> >>>>>>ls -l
> >>>>>>/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages/easybuild_framework-2.8.1-py2.7.egg/easybuild/__init__.py
> >>>>>Yes
> >>>>>
> >>>>>>Sorry, I meant:
> >>>>>>
> >>>>>> cat
> >>>>>> /opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages/easy-install.pth
> >>>>>import sys; sys.__plen = len(sys.path)
> >>>>>./distribute-0.7.3-py2.7.egg
> >>>>>./setuptools-22.0.2-py2.7.egg
> >>>>>./vsc_install-0.10.6-py2.7.egg
> >>>>>./vsc_base-2.5.1-py2.7.egg
> >>>>>./easybuild_framework-2.8.1-py2.7.egg
> >>>>>./easybuild_easyblocks-2.8.1-py2.7.egg
> >>>>>./easybuild_easyconfigs-2.8.1-py2.7.egg
> >>>>>import sys; new = sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p =
> >>>>>getattr(sys, '__egginsert', 0); sys.path[p:p] = new; sys.__egginsert = p
> >>>>>+ len(new)
> >>>>Looks fine...
> >>>>>>and let's throw in this too:
> >>>>>>
> >>>>>> cat
> >>>>>> /opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages/setuptools.pth
> >>>>>./setuptools-22.0.2-py2.7.egg
> >>>>This is fine too.
> >>>>>Any PYTHONPATH issue?
> >>>>>I tried unsetenv PYTHONPATH but doid not change anything
> >>>> From the log you provided, right before the "/usr/bin/python -c
> >>>>'import easybuid' " that fails.
> >>>>
> >>>>== 2016-06-03 16:53:19,262 environment.py:97 INFO Environment
> >>>>variable PYTHONPATH set to
> >>>>/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages:/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib64/python2.7/site-packages
> >>>>(previous value: ...)
> >>>>
> >>>>So, looks fine. I have no idea why the import fails...
> >>>>
> >>>>OK, let's dig a little deeper.
> >>>>
> >>>>Can you rebootstrap with debugging enabled, and send back the full log?
> >>>>
> >>>> export EASYBUILD_BOOTSTRAP_DEBUG=1
> >>>> python bootstrap_eb.py /opt/local/easybuild-2.8.1 2>&1 | tee
> >>>>eb-debug.log
> >>>>
> >>>>Also, let's try this (after the failing bootstrap):
> >>>>
> >>>>PYTHONPATH=/opt/local/easybuild-2.8.1/software/Core/EasyBuild/2.8.1/lib/python2.7/site-packages
> >>>>python -c "import easybuild"
> >>>>
> >>>>
> >>>>regards,
> >>>>
> >>>>Kenneth
> >>>>
> >>>>
> >>>>>Bernd
> >>>>>
> >>>>>>regards,
> >>>>>>
> >>>>>>Kenneth
> >>>>>>>;-)
> >>>>>>>
> >>>>>>>Bernd
> >>>>>>>
> >>>>>>>>regards,
> >>>>>>>>
> >>>>>>>>Kenneth
> >>>>>>>>
> >>>>>>>>On 03/06/16 17:00, Bernd Mohr wrote:
> >>>>>>>>>Fresh install with
> >>>>>>>>>
> >>>>>>>>>% ./bootstrap_eb.py /opt/local/easybuild-2.8.1
> >>>>>>>>>
> >>>>>>>>>on a fresh (empty) SuSE LEAP 42.1 system.
> >>>>>>>>>
> >>>>>>>>>% /usr/bin/python --version
> >>>>>>>>>Python 2.7.9
> >>>>>>>>>
> >>>>>>>>>Failure in Stage2:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>+++ STAGE 2: installing EasyBuild in /opt/local/easybuild-2.8.1 with
> >>>>>>>>>EasyBuild from stage 1...
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>== temporary log file in case of crash
> >>>>>>>>>/tmp/eb-ZufCws/easybuild-bC72PS.log
> >>>>>>>>>== processing EasyBuild easyconfig /tmp/tmpnlyQGk/EasyBuild-2.8.1.eb
> >>>>>>>>>== building and installing Core/EasyBuild/2.8.1...
> >>>>>>>>>== fetching files...
> >>>>>>>>>== creating build dir, resetting environment...
> >>>>>>>>>== unpacking...
> >>>>>>>>>== patching...
> >>>>>>>>>== preparing...
> >>>>>>>>>== configuring...
> >>>>>>>>>== building...
> >>>>>>>>>== testing...
> >>>>>>>>>== installing...
> >>>>>>>>>== taking care of extensions...
> >>>>>>>>>== postprocessing...
> >>>>>>>>>== sanity checking...
> >>>>>>>>>== FAILED: Installation ended unsuccessfully (build directory:
> >>>>>>>>>/tmp/tmpnlyQGk/EasyBuild/2.8.1/dummy-dummy): build failed (first 300
> >>>>>>>>>chars): Sanity check failed: EasyBuild failed to install, cmd
> >>>>>>>>>'/usr/bin/python -c "import easybuild"' (stdin: None) output:
> >>>>>>>>>Traceback (most recent call last):
> >>>>>>>>> File "<string>", line 1, in <module>
> >>>>>>>>>ImportError: No module named easybuild
> >>>>>>>>>, sanity check command eb --version exited with code 1 (output:
> >>>>>>>>>/usr/bi
> >>>>>>>>>
> >>>>>>>>>Any ideas?
> >>>>>>>>>
> >>>>>>>>>Full installation log attached.
> >>>>>>>>>
> >>>>>>>>>Bernd
> >>>>>>>>>
> >>>>>>>>>--
> >>>>>>>>>Dr.-Ing. Bernd Mohr
> >>>>>>>>>Juelich Supercomputing Centre
> >>>>>>>>>Institute for Advanced Simulation
> >>>>>>>>>
> >>>>>>>>>E-Mail: [email protected]
> >>>>>>>>>WWW:
> >>>>>>>>>http://www.fz-juelich.de/SharedDocs/Personen/IAS/JSC/EN/staff/mohr_b.html
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>------------------------------------------------------------------------------------------------
> >>>>>>>>>------------------------------------------------------------------------------------------------
> >>>>>>>>>Forschungszentrum Juelich GmbH
> >>>>>>>>>52425 Juelich
> >>>>>>>>>Sitz der Gesellschaft: Juelich
> >>>>>>>>>Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> >>>>>>>>>Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> >>>>>>>>>Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> >>>>>>>>>Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> >>>>>>>>>Prof. Dr. Sebastian M. Schmidt
> >>>>>>>>>------------------------------------------------------------------------------------------------
> >>>>>>>>>------------------------------------------------------------------------------------------------
> >>>>>>>>>
--
Dr.-Ing. Bernd Mohr
Juelich Supercomputing Centre
Institute for Advanced Simulation
E-Mail: [email protected]
WWW:
http://www.fz-juelich.de/SharedDocs/Personen/IAS/JSC/EN/staff/mohr_b.html
