Hi Cormac, On 24 Dec 2013, at 18:29, Garvey, Cormac T wrote:
> Hi Kenneth, > Attached is the gzipped Debug log file. I think I found the problem... The numpy easyblock removes the numpy build directory to avoid scipy finding it later (this causes unwanted behavior), see the install_step implementation in https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/n/numpy.py . However, no chdir is done after removing that directory... It seems that we somehow end up in that directory after doing the numpy install, which is probably causing Lmod to fail (it doesn't like to be run in a non-existing directory). I'm still verifying whether this is really the issue, I should be able to easily reproduce this on my end if this is indeed the case. In any case, https://github.com/hpcugent/easybuild-easyblocks/pull/345 should fix this. The diff in that PR looks a bit big because I moved up the install_step function (it should be above sanity_check_step, style-wise). The only important line is the "os.chdir(self.builddir)" after the "rmtree2" call. Let us know if that helps (preferably in the PR itself, so it can be merged in). regards, Kenneth > > Thanks, > cormac. > > > On Mon, Dec 23, 2013 at 1:59 PM, Kenneth Hoste <[email protected]> wrote: > Hi Cormac, > > On 23 Dec 2013, at 18:37, Garvey, Cormac T wrote: > > > Hi Kenneth > > No problem, sorry for pulling you away from other tasks. > > > > Yes, I am using lmod 5.2 > > I've tried to reproduce your problem on my end, but no luck, the build is > working fine for me even if I use Lmod. I'm using EasyBuild v1.9.0 and Lmod > v5.2 as well. > > Can you please provide a full debug log of the failing Python + numpy build > (just gzip the raw log file, and include it as attachment in reply, should be > small enough)? > I'm missing some context to figure out what's going on, a full debug log > should help a lot. > > The first log message you showed (see below) tells me more is going on before > the part you pasted, i.e. the real problem is happening earlier... > > > == 2013-12-18 14:39:49,745 main.filetools WARNING Found 5 errors in command > > output (output: collect2: error: ld returned 1 exit status, collect2: > > error: ld returned 1 exit status, collect2: error: ld returned 1 exit > > status, collect2: error: ld returned 1 exit status, collect2: error: ld > > returned 1 exit status) > > > regards, > > Kenneth > > > > > > Merry Christmas, > > Cormac. > > > > > > On Mon, Dec 23, 2013 at 10:33 AM, Kenneth Hoste <[email protected]> > > wrote: > > Hi Cormac, > > > > Sorry for not answering this earlier... > > > > I'll try and reproduce this behavior to see what the problem is. > > > > It seems that Lmod is being called from a non-existing directory, which it > > doesn't like... Shouldn't be too hard to fix, I expect (if I can track down > > where this is happening exactly). > > > > Which Lmod version are you using here? The latest Lmod v5.2? > > > > > > regards, > > > > Kenneth > > > > On 23 Dec 2013, at 17:31, Garvey, Cormac T wrote: > > > > > Hi All, > > > When I build Python + numpy with lmod disabled, it builds correctly. It > > > looks like a problem building python (with extensions) with > > > lmod enabled. > > > > > > Thanks, > > > Cormac. > > > > > > > > > On Wed, Dec 18, 2013 at 5:17 PM, Garvey, Cormac T <[email protected]> > > > wrote: > > > > > > Also, note when I build numpy as a separate module, it builds correctly. > > > > > > numpy easyconfig > > > > > > > > > name = 'numpy' > > > version = '1.7.1' > > > python = 'Python' > > > pyver = '2.7.5' > > > versionsuffix = '-%s-%s' % (python, pyver) > > > > > > homepage = 'http://www.numpy.org' > > > description = """NumPy is the fundamental package for scientific > > > computing with Python. It contains among other things: > > > a powerful N-dimensional array object, sophisticated (broadcasting) > > > functions, tools for integrating C/C++ and Fortran > > > code, useful linear algebra, Fourier transform, and random number > > > capabilities. Besides its obvious scientific uses, > > > NumPy can also be used as an efficient multi-dimensional container of > > > generic data. Arbitrary data-types can be > > > defined. This allows NumPy to seamlessly and speedily integrate with a > > > wide variety of databases.""" > > > > > > > > > toolchain = {'name': 'gnupack', 'version': '1.1.1'} > > > > > > source_urls = [('http://sourceforge.net/projects/numpy/files/NumPy/%s' % > > > version, 'download')] > > > sources = ['%s-%s.tar.gz' % (name, version)] > > > > > > nose = 'nose' > > > nosever = '1.3.0' > > > > > > builddependencies = [ > > > (nose,nosever,'-%s-%s' % (python,pyver)), > > > ] > > > > > > dependencies = [ > > > (python, pyver), > > > ] > > > > > > moduleclass = 'math' > > > > > > > > > > > > Thanks, > > > Cormac. > > > > > > > > > > > > On Wed, Dec 18, 2013 at 4:17 PM, Garvey, Cormac T <[email protected]> > > > wrote: > > > > > > Hi All, > > > I am having some difficult build python 2.7.5 with Easybuild 1.9.0 + lmod > > > > > > My easyconfig file looks like > > > > > > name = 'Python' > > > version = '2.7.5' > > > > > > homepage = 'http://python.org/' > > > description = "Python is a programming language that lets you work more > > > quickly and integrate your systems more effectively." > > > > > > toolchain = {'name': 'gnupack', 'version': '1.1.1'} > > > toolchainopts = {'pic': True, 'opt': True, 'optarch': True} > > > > > > numpyversion = '1.7.1' > > > > > > source_urls = ['http://www.python.org/ftp/%s/%s/' % (name.lower(), > > > version)] > > > sources = ['%s-%s.tgz' % (name, version)] > > > > > > patches = ['python-%(version)s_libffi_int128_icc.patch'] > > > > > > exts_list = [ > > > ('nose', '1.3.0', { > > > 'source_urls': ['http://pypi.python.org/packages/source/n/nose/'], > > > }), > > > ('numpy', numpyversion, { > > > 'source_urls': [('http://sourceforge.net/projects/numpy/files/NumPy/%s' > > > % numpyversion, 'download')], > > > 'patches': [ > > > 'numpy-%s_distutils_multiple-lib-dirs.patch' % numpyversion, > > > 'numpy-iqacml.patch', > > > ], > > > }), > > > ] > > > > > > moduleclass = 'lang' > > > > > > > > > > > > With debugging turned on (EASYBUILD_DEBUG=1), I get an error building the > > > module numpy. > > > > > > == 2013-12-18 14:39:49,745 main.filetools WARNING Found 5 errors in > > > command output (output: collect2: error: ld returned 1 exit status, > > > collect2: error: ld returned 1 exit status, collect2: error: ld returned > > > 1 exit status, collect2: error: ld returned 1 exit status, collect2: > > > error: ld returned 1 exit status) > > > == 2013-12-18 14:39:49,745 main.environment INFO Environment variable > > > PYTHONPATH set to > > > /hpc-common/software/easybuild/inl-easyconfigs:/hpc-common/software/easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages > > > == 2013-12-18 14:39:50,759 main.filetools INFO Path > > > /apps/local/easybuild/build/Python/2.7.5/gnupack-1.1.1/numpy successfully > > > removed. > > > == 2013-12-18 14:39:50,761 main.Lmod INFO $MODULEPATH set based on list > > > of module paths: > > > /tmp/tmpT3IMq2/all:/apps/local/easybuild/modules/all:/apps/local/modules/modulefiles:/apps/projects/modules/modulefiles:/apps/local/modules/modulefiles/Linux:/apps/local/modules/modulefiles/Core > > > == 2013-12-18 14:39:50,762 main.filetools INFO Command lmod found at > > > /apps/projects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod > > > == 2013-12-18 14:39:50,762 main.Lmod INFO Full path for module command is > > > /apps/projects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod, so using it > > > == 2013-12-18 14:39:50,763 main.Lmod ERROR EasyBuild crashed with an > > > error (at > > > easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages/easybuild/tools/modules.py:625 > > > in __init__): Failed to check Lmod version: [Errno 2] No such file or > > > directory > > > > > > The exception is occurring at the line below (file modules.py, function > > > run_module), when it tries to execute os.getcwd()? > > > > > > self.log.debug("Running module command '%s' from %s" % (' '.join(cmdlist > > > + args), os.getcwd())) > > > > > > > > > When I turn debugging-off > > > > > > I get the following error > > > > > > == 2013-12-18 16:13:28,127 main.filetools WARNING Found 5 errors in > > > command output (output: collect2: error: ld returned 1 exit status, > > > collect2: error: ld returned 1 exit status, collect2: error: ld returned > > > 1 exit status, collect2: error: ld returned 1 exit status, collect2: > > > error: ld returned 1 exit status) > > > == 2013-12-18 16:13:28,128 main.environment INFO Environment variable > > > PYTHONPATH set to > > > /hpc-common/software/easybuild/inl-easyconfigs:/hpc-common/software/easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages > > > == 2013-12-18 16:13:29,178 main.filetools INFO Path > > > /apps/local/easybuild/build/Python/2.7.5/gnupack-1.1.1/numpy successfully > > > removed. > > > == 2013-12-18 16:13:29,181 main.Lmod INFO $MODULEPATH set based on list > > > of module paths: > > > /tmp/tmpv_sTqF/all:/apps/local/easybuild/modules/all:/apps/local/modules/modulefiles:/apps/projects/modules/modulefiles:/apps/local/modules/modulefiles/Linux:/apps/local/modules/modulefiles/Core > > > == 2013-12-18 16:13:29,182 main.filetools INFO Command lmod found at > > > /apps/projects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod > > > == 2013-12-18 16:13:29,182 main.Lmod INFO Full path for module command is > > > /apps/projects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod, so using it > > > == 2013-12-18 16:13:29,182 main.Lmod INFO Before module update > > > == 2013-12-18 16:13:29,269 main.Lmod INFO After module update > > > == 2013-12-18 16:13:29,270 main.Lmod INFO After tempfile > > > == 2013-12-18 16:13:29,270 main.Lmod INFO Before open > > > == 2013-12-18 16:13:29,270 main.Lmod INFO After open > > > == 2013-12-18 16:13:29,355 main.Lmod INFO After subprocess > > > == 2013-12-18 16:13:29,356 main.Lmod INFO After open > > > == 2013-12-18 16:13:29,356 main.Lmod INFO After read > > > == 2013-12-18 16:13:29,356 main.Lmod INFO lmod help output: > > > /apps/projects/lua/5.1.4.5/gcc-4.5.1/opt/bin/lua: > > > ...gcc-4.5.1/opt/lmod/lmod/libexec/../tools/fileOps.lua:295: bad argument > > > #1 to 'chdir' (string expected, got nil) > > > stack traceback: > > > [C]: in function 'chdir' > > > ...gcc-4.5.1/opt/lmod/lmod/libexec/../tools/fileOps.lua:295: in > > > function 'abspath' > > > ...s/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/utils.lua:427: in > > > function 'readRC' > > > ...ojects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod:472: in > > > function 'main' > > > ...ojects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod:618: in main > > > chunk > > > [C]: ? > > > > > > > > > Any suggestions? > > > > > > Thanks, > > > Cormac. > > > > > > > > > > > > -- > > > Cormac Garvey > > > HPC Software Consultant > > > Scientific Computing > > > Idaho National Laboratory > > > Ph: 208-526-6294 > > > > > > > > > > > > > > > -- > > > Cormac Garvey > > > HPC Software Consultant > > > Scientific Computing > > > Idaho National Laboratory > > > Ph: 208-526-6294 > > > > > > > > > > > > > -- > > Cormac Garvey > > HPC Software Consultant > > Scientific Computing > > Idaho National Laboratory > > Ph: 208-526-6294 > > > > > > > -- > Cormac Garvey > HPC Software Consultant > Scientific Computing > Idaho National Laboratory > Ph: 208-526-6294 > > <easybuild-rsUBi8.log.gz>
