Howdy Ken, Attached is a demo/trace.
Pretty sure it isn't .pyc files since:
a) I disable Python compiles[*] because...
b) .pyc's between x86_64 and ppc64 aren't compatible
[*] except when using the EB boostrap script... it gets _very_
unhappy if I set PYTHONDONTWRITEBYTECODE... but that
is only during bootstrap
Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
[email protected] http://sc.tamu.edu
SC Helpdesk: [email protected]
On 05/27/2015 03:07 AM, Kenneth Hoste wrote:
Hi Jack, (feedback below) On 26/05/15 21:41, Jack Perdue wrote:I've seen very weird behavior with symlinks and/or PYTHONPATH. I have directories like this (easier to cut/paste than to explain): $ tree -L 1 /general/software/noarch/easybuild/ /general/software/noarch/easybuild/ |-- ebfiles_repo |-- ebfiles_repo_cleaned |-- sources `-- tamusc $ tree -L 1 /general/software/x86_64/easybuild/ /general/software/x86_64/easybuild/ |-- ebfiles_repo -> /general/software/noarch/easybuild/ebfiles_repo|-- ebfiles_repo_cleaned -> /general/software/noarch/easybuild/ebfiles_repo_cleaned|-- modulefiles |-- modules |-- software |-- sources -> /general/software/noarch/easybuild/sources |-- tamusc `-- Westmere $ tree -L 1 /general/software/x86_64/easybuild/tamusc/ /general/software/x86_64/easybuild/tamusc/ |-- bin -> /general/software/noarch/easybuild/tamusc/bin |-- easybuild -> /general/software/noarch/easybuild/tamusc/easybuild |-- ec-dev -> /general/software/noarch/easybuild/tamusc/ec-dev |-- etc -> /general/software/noarch/easybuild/tamusc/etc |-- githubs -> /general/software/noarch/easybuild/tamusc/githubs `-- python_virtenv I have similar trees for ppc64. In practice, our tamusc EB directory is available at /software/easybuild/tamusc (more symlink magic). The easybuild directory in the last listing is where all my development python files are. These are not accessed via PYTHONPATH, but instead are utilized through a set of sym-links in my development EB install (overriding framework files via PYTHONPATH is nearly impossible). Now then, I typically do my builds in ec-dev, which is on the same level as the easybuild directory. PYTHONPATH points to the installed copy of EB, but on x86_64, I have NOT overridden any symlinks with pointers to the tamusc/easybuild directory (that is mostly done on ppc64). However, of late (since 2.0 maybe), I can not do a build in the ec-dev directory on x86_64 because it somehow finds the modifications made for ppc64 and I get an error like: ImportError: No module named xlcxlf (where xlcxlf is the IBM compilers on ppc64). I can find no logical/reasonable explanation for eb to find any files that reference xlcxlf on x86_64. Note that if I go one directory deeper, e.g. "ec-dev/ada", then the problem goes away. So it would seem that at some point, EasyBuild is seeing the sibling "easybuild" directory when building in "ec-dev" and gets broken by ppc64 modifications. When I go deeper and "easybuild" is no longer a sibling of the current directory, it works. Very strange... I got no ideas.Interesting setup you've got there...A likely cause for this are .pyc files that cache the compiled Python bytecode, specifically those for the __init__.py files that 'register' Python packages.I'm not sure, but it's possible that when you're using EasyBuild in the ppc64 directory, .pyc files in other locations get updated, and you're unwillingly putting points in place to paths in the ppc64 workspace. Maybe you can check by running 'strings' on the __init__.pyc files in your x86_64 directory?Do you get a traceback from Python along with the ImportError you run into, to figure out where the ImportError is happening exactly?I suspect it's happening via the 'search_toolchain' function (in easybuild/tools/toolchain/utilities.py), that is used at startup time to figure out which toolchains are known.regards, Kennethjack On 05/26/2015 02:13 PM, Kenneth Hoste wrote:On 26/05/15 17:46, Mohr, Bernd wrote:Yes, I can reproduce the issue. See attached typescriptIs this $HOME/easyblocks path used in your environment somehow? $PYTHONPATH maybe?This is very weird... :-) K.Bernd Sent from Windows Mail *From:* Kenneth Hoste <mailto:[email protected]> *Sent:* Tuesday, 26 May, 2015 16:18 *To:* EasyBuild <mailto:[email protected]> Hi Bernd, On 26/05/15 12:29, Mohr, Bernd wrote: Kenneth, by accident, I found the cause for the failure In my home directory I had created a symbolic link easyblocks →/opt/local/easybuild-2.1/software/Core/EasyBuild/2.1.0/lib/python2.7/site-packages/easybuild_easyblocks-2.1.0-py2.7.egg/easybuild/easyblocks/to be able to quicker access the code. Of course, I was running the bootstrap script from my home directory 😉 All fine now.As far as I can tell, having a symlink like that present shouldn't impact the bootstrap procedure at all...Is that the only thing you changed? And can you reproduce the problem now if you put the symlink back in place (to the 2.1.0 directory)?regards, Kenneth Thanks Bernd *From:* Kenneth Hoste <mailto:[email protected]> *Sent:* Tuesday, 26 May, 2015 11:42 *To:* EasyBuild <mailto:[email protected]> Hi Bernd, On 26/05/15 11:32, Mohr, Bernd wrote: Dear EasyBuild team, I tried to bootstrap v.2.1.1 (method (i)), but get the following error:**********************************************************************************************************************carbon-vm:~ [103] ./bootstrap_eb.py --prefix=/opt/local/easybuild-2.1.1 Don't use --prefix here, just specify the install prefix to bootstrap EasyBuild into directly: $ ./bootstrap_eb.py /opt/local/easybuild-2.1.1 This is not the cause of the problem however, since this only applies to stage 2 of the bootstrap... [[INFO]] Found module command 'lmod' (Lmod), so using it. [[INFO]] +++ STAGE 0: installing distribute via included (patched) distribute_setup.py... Downloadinghttp://pypi.python.org/packages/source/d/distribute/distribute-0.6.34.tar.gzExtracting in /tmp/tmp8TWdBC Now working in /tmp/tmp8TWdBC/distribute-0.6.34 Installing Distribute [[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install... [[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmpvL8XTk/eb_stage1 easybuild' Installing with setuptools.setup... Installing version 2.1.1 warning: install_lib: 'build/lib' does not exist -- no Python modules to install zip_safe flag not set; analyzing archive contents... Installing with setuptools.setup... Installing version 2.1.1 (API version 2) Installing with setuptools.setup... Installing version 2.1.0 (required versions: API >= 2) error: Could not find required distribution easybuild-easyblocks==2.1.1 For some reason, easy_install is deciding to go with easybuild-easyblocks v2.1.0, only to then complain that it can't find v2.1.1 ... I've just retried myself, I can't reproduce this problem; bootstrap is working fine for me (just like it did when I tested this after the release). Can you provide debug output by defining $EASYBUILD_BOOTSTRAP_DEBUG to something, e.g. "export EASYBUILD_BOOTSTRAP_DEBUG=1" ? Maybe also share what you have in $HOME/.local/lib/python2.*/site-packages, if anything. regards, Kenneth**********************************************************************************************************************Any idea what is wrong? Bernd *From:* Kenneth Hoste <mailto:[email protected]> *Sent:* Monday, 18 May, 2015 19:20 *To:* EasyBuild <mailto:[email protected]> Hello EasyBuilders, I'm happy to announce the release of EasyBuild v2.1.1, which is the best release so far, no doubt about it. (*) This is a bugfix release, mainly motivated by some issues with the --module-only command line option that was introduced in EasyBuild v2.1.0: modules generated via --module-only in combination with --force were missing 'module load' statements for dependencies, and several easyblocks turned out to be incompatible with --module-only. These problems have been resolved, and the unit tests were enhanced to try and make sure easyblocks remain compatible with --module-only. Other changes include support for supplying regular expression patterns to --search (and only considering the easyconfig filename, rather than the entire path, see [2]), support for builing and installing GROMACS v5.x, adding several new toolchain versions (ictce/7.3.5 and Cray*), and support for installing 3 new software packages. A detailed overview of all changes is available in the EasyBuild release notes [3]. To upgrade to EasyBuild v2.1.1, there are several options: (i) (re)bootstrap EasyBuild to obtain an EasyBuild/2.1.1 module to load [4] (ii) install EasyBuild v2.1.1 with a previous version of EasyBuild, using the easyconfig file available in [5] note: make sure you're using https PyPI source URLs in case you're using an easyconfig file obtained via some other way, cfr. [6] (iii) install EasyBuild v2.1.1 from PyPI, using one of the standard Python installation tools (easy_install, pip, ...) [7] (iv) updating your Git working copies of the different EasyBuild repositories Enjoy! regards, Kenneth EasyBuild release manager [1] https://pypi.python.org/pypi/easybuild [2]https://easybuild.readthedocs.org/en/latest/Using_the_EasyBuild_command_line.html#searching-for-easyconfigs[3] https://easybuild.readthedocs.org/en/latest/Release_notes.html [4]https://easybuild.readthedocs.org/en/latest/Installation.html#bootstrapping-easybuild [5] https://github.com/hpcugent/easybuild-easyconfigs/pull/1615[6] https://github.com/hpcugent/easybuild-framework/pull/1286 [7]https://easybuild.readthedocs.org/en/latest/Installation_Alternative.html#standard-installation-of-latest-release(*): I'm aware I stated the same about EasyBuild v2.1.0, but this was a valid statement at the time.------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen HuthmacherGeschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
[j-perdue@ada2 ~]$ cd /software/easybuild/tamusc/ec-dev/
[j-perdue@ada2 ec-dev]$ module load EasyBuild-tamusc-ada
[j-perdue@ada2 ada]$ ml
Currently Loaded Modules:
1) EasyBuild/2.1.0-tamusc 2) EasyBuild-tamusc-ada/2.1.0
[j-perdue@ada2 ec-dev]$ env | grep EASY | sort
EASYBUILD_BUILDPATH=/tmp/easybuild
EASYBUILD_DEPRECATED=2.0
EASYBUILD_MODULES_TOOL=Lmod
EASYBUILD_PREFIX=/software/easybuild
EASYBUILD_REPOSITORYPATH=/software/easybuild/ebfiles_repo/ada
EASYBUILD_ROBOT=/software/easybuild/ebfiles_repo_cleaned/ada
EASYBUILD_SOURCEPATH=/software/easybuild/sources
EBDEVELEASYBUILD=/software/easybuild/software/EasyBuild/2.1.0-tamusc/easybuild/EasyBuild-2.1.0-tamusc-easybuild-devel
EBROOTEASYBUILD=/software/easybuild/software/EasyBuild/2.1.0-tamusc
EBVERSIONEASYBUILD=2.1.0
__LMOD_STACK_EASYBUILD_BUILDPATH=L3RtcC9lYXN5YnVpbGQ=
__LMOD_STACK_EASYBUILD_DEPRECATED=Mi4w
__LMOD_STACK_EASYBUILD_MODULES_TOOL=TG1vZA==
__LMOD_STACK_EASYBUILD_PREFIX=L3NvZnR3YXJlL2Vhc3lidWlsZA==
__LMOD_STACK_EASYBUILD_REPOSITORYPATH=L3NvZnR3YXJlL2Vhc3lidWlsZC9lYmZpbGVzX3JlcG8vYWRh
__LMOD_STACK_EASYBUILD_ROBOT=L3NvZnR3YXJlL2Vhc3lidWlsZC9lYmZpbGVzX3JlcG9fY2xlYW5lZC9hZGE=
__LMOD_STACK_EASYBUILD_SOURCEPATH=L3NvZnR3YXJlL2Vhc3lidWlsZC9zb3VyY2Vz
[j-perdue@ada2 ec-dev]$ find $EBROOTEASYBUILD -name \*.pyc | xargs rm
[j-perdue@ada2 ec-dev]$ find ../easybuild -name \*.pyc | xargs rm
rm: missing operand
Try `rm --help' for more information.
[j-perdue@ada2 ec-dev]$ env | grep PYTHON
__LMOD_STACK_PYTHONDONTWRITEBYTECODE=Tk9fUFlDU19GT1JfQVJDSF9JTkRFUEVOREVOQ0U=
PYTHONDONTWRITEBYTECODE=NO_PYCS_FOR_ARCH_INDEPENDENCE
PYTHONPATH=/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages
[j-perdue@ada2 ec-dev]$ eb zlib-1.2.5-ictce-6.2.5.eb
== temporary log file in case of crash /tmp/eb-rvkF9S/easybuild-TTA8ft.log
Traceback (most recent call last):
File
"/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages/easybuild_framework-2.1.0-py2.6.egg/easybuild/main.py",
line 339, in <module>
main()
File
"/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages/easybuild_framework-2.1.0-py2.6.egg/easybuild/main.py",
line 254, in main
easyconfigs, generated_ecs = parse_easyconfigs(paths)
File
"/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages/easybuild_framework-2.1.0-py2.6.egg/easybuild/framework/easyconfig/tools.py",
line 342, in parse_easyconfigs
ecs = process_easyconfig(ec_file, **kwargs)
File
"/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages/easybuild_framework-2.1.0-py2.6.egg/easybuild/framework/easyconfig/easyconfig.py",
line 976, in process_easyconfig
if ec.toolchain.name != DUMMY_TOOLCHAIN_NAME:
File
"/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages/easybuild_framework-2.1.0-py2.6.egg/easybuild/framework/easyconfig/easyconfig.py",
line 460, in toolchain
self._toolchain = get_toolchain(self['toolchain'], self['toolchainopts'],
mns=ActiveMNS())
File
"/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages/easybuild_framework-2.1.0-py2.6.egg/easybuild/tools/toolchain/utilities.py",
line 125, in get_toolchain
tc_class, all_tcs = search_toolchain(tc['name'])
File
"/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages/easybuild_framework-2.1.0-py2.6.egg/easybuild/tools/toolchain/utilities.py",
line 66, in search_toolchain
tc_modules = import_available_modules('easybuild.toolchains')
File
"/software/easybuild/software/EasyBuild/2.1.0-tamusc/lib/python2.6/site-packages/easybuild_framework-2.1.0-py2.6.egg/easybuild/tools/utilities.py",
line 102, in import_available_modules
modules.append(__import__(modpath, globals(), locals(), ['']))
ImportError: No module named xlcxlf
[j-perdue@ada2 ec-dev]$ cd ada/
[j-perdue@ada2 ada]$ eb ../zlib-1.2.5-ictce-6.2.5.eb
== temporary log file in case of crash /tmp/eb-kxwQbu/easybuild-LftdIU.log
== resolving dependencies ...
== processing EasyBuild easyconfig
/general/software/noarch/easybuild/tamusc/ec-dev/zlib-1.2.5-ictce-6.2.5.eb
== building and installing zlib/1.2.5-ictce-6.2.5...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
[snipped]
