For the record: problem solved via PR https://github.com/hpcugent/easybuild-easyblocks/pull/345 that was merged in just now.
Thanks for reporting the issue and reviewing the PR Cormac! regards, Kenneth On 02 Jan 2014, at 08:49, Kenneth Hoste wrote: > Hi Cormac, > > On 24 Dec 2013, at 18:29, Garvey, Cormac T wrote: > >> Hi Kenneth, >> Attached is the gzipped Debug log file. > > I think I found the problem... > > The numpy easyblock removes the numpy build directory to avoid scipy finding > it later (this causes unwanted behavior), see the install_step implementation > in > https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/n/numpy.py > . > > However, no chdir is done after removing that directory... > It seems that we somehow end up in that directory after doing the numpy > install, which is probably causing Lmod to fail (it doesn't like to be run in > a non-existing directory). > > I'm still verifying whether this is really the issue, I should be able to > easily reproduce this on my end if this is indeed the case. > > In any case, https://github.com/hpcugent/easybuild-easyblocks/pull/345 should > fix this. > The diff in that PR looks a bit big because I moved up the install_step > function (it should be above sanity_check_step, style-wise). > The only important line is the "os.chdir(self.builddir)" after the "rmtree2" > call. > > Let us know if that helps (preferably in the PR itself, so it can be merged > in). > > > > regards, > > Kenneth > > >> >> Thanks, >> cormac. >> >> >> On Mon, Dec 23, 2013 at 1:59 PM, Kenneth Hoste <[email protected]> >> wrote: >> Hi Cormac, >> >> On 23 Dec 2013, at 18:37, Garvey, Cormac T wrote: >> >>> Hi Kenneth >>> No problem, sorry for pulling you away from other tasks. >>> >>> Yes, I am using lmod 5.2 >> >> I've tried to reproduce your problem on my end, but no luck, the build is >> working fine for me even if I use Lmod. I'm using EasyBuild v1.9.0 and Lmod >> v5.2 as well. >> >> Can you please provide a full debug log of the failing Python + numpy build >> (just gzip the raw log file, and include it as attachment in reply, should >> be small enough)? >> I'm missing some context to figure out what's going on, a full debug log >> should help a lot. >> >> The first log message you showed (see below) tells me more is going on >> before the part you pasted, i.e. the real problem is happening earlier... >> >>> == 2013-12-18 14:39:49,745 main.filetools WARNING Found 5 errors in command >>> output (output: collect2: error: ld returned 1 exit status, collect2: >>> error: ld returned 1 exit status, collect2: error: ld returned 1 exit >>> status, collect2: error: ld returned 1 exit status, collect2: error: ld >>> returned 1 exit status) >> >> >> regards, >> >> Kenneth >> >> >>> >>> Merry Christmas, >>> Cormac. >>> >>> >>> On Mon, Dec 23, 2013 at 10:33 AM, Kenneth Hoste <[email protected]> >>> wrote: >>> Hi Cormac, >>> >>> Sorry for not answering this earlier... >>> >>> I'll try and reproduce this behavior to see what the problem is. >>> >>> It seems that Lmod is being called from a non-existing directory, which it >>> doesn't like... Shouldn't be too hard to fix, I expect (if I can track down >>> where this is happening exactly). >>> >>> Which Lmod version are you using here? The latest Lmod v5.2? >>> >>> >>> regards, >>> >>> Kenneth >>> >>> On 23 Dec 2013, at 17:31, Garvey, Cormac T wrote: >>> >>>> Hi All, >>>> When I build Python + numpy with lmod disabled, it builds correctly. It >>>> looks like a problem building python (with extensions) with >>>> lmod enabled. >>>> >>>> Thanks, >>>> Cormac. >>>> >>>> >>>> On Wed, Dec 18, 2013 at 5:17 PM, Garvey, Cormac T <[email protected]> >>>> wrote: >>>> >>>> Also, note when I build numpy as a separate module, it builds correctly. >>>> >>>> numpy easyconfig >>>> >>>> >>>> name = 'numpy' >>>> version = '1.7.1' >>>> python = 'Python' >>>> pyver = '2.7.5' >>>> versionsuffix = '-%s-%s' % (python, pyver) >>>> >>>> homepage = 'http://www.numpy.org' >>>> description = """NumPy is the fundamental package for scientific computing >>>> with Python. It contains among other things: >>>> a powerful N-dimensional array object, sophisticated (broadcasting) >>>> functions, tools for integrating C/C++ and Fortran >>>> code, useful linear algebra, Fourier transform, and random number >>>> capabilities. Besides its obvious scientific uses, >>>> NumPy can also be used as an efficient multi-dimensional container of >>>> generic data. Arbitrary data-types can be >>>> defined. This allows NumPy to seamlessly and speedily integrate with a >>>> wide variety of databases.""" >>>> >>>> >>>> toolchain = {'name': 'gnupack', 'version': '1.1.1'} >>>> >>>> source_urls = [('http://sourceforge.net/projects/numpy/files/NumPy/%s' % >>>> version, 'download')] >>>> sources = ['%s-%s.tar.gz' % (name, version)] >>>> >>>> nose = 'nose' >>>> nosever = '1.3.0' >>>> >>>> builddependencies = [ >>>> (nose,nosever,'-%s-%s' % (python,pyver)), >>>> ] >>>> >>>> dependencies = [ >>>> (python, pyver), >>>> ] >>>> >>>> moduleclass = 'math' >>>> >>>> >>>> >>>> Thanks, >>>> Cormac. >>>> >>>> >>>> >>>> On Wed, Dec 18, 2013 at 4:17 PM, Garvey, Cormac T <[email protected]> >>>> wrote: >>>> >>>> Hi All, >>>> I am having some difficult build python 2.7.5 with Easybuild 1.9.0 + lmod >>>> >>>> My easyconfig file looks like >>>> >>>> name = 'Python' >>>> version = '2.7.5' >>>> >>>> homepage = 'http://python.org/' >>>> description = "Python is a programming language that lets you work more >>>> quickly and integrate your systems more effectively." >>>> >>>> toolchain = {'name': 'gnupack', 'version': '1.1.1'} >>>> toolchainopts = {'pic': True, 'opt': True, 'optarch': True} >>>> >>>> numpyversion = '1.7.1' >>>> >>>> source_urls = ['http://www.python.org/ftp/%s/%s/' % (name.lower(), >>>> version)] >>>> sources = ['%s-%s.tgz' % (name, version)] >>>> >>>> patches = ['python-%(version)s_libffi_int128_icc.patch'] >>>> >>>> exts_list = [ >>>> ('nose', '1.3.0', { >>>> 'source_urls': ['http://pypi.python.org/packages/source/n/nose/'], >>>> }), >>>> ('numpy', numpyversion, { >>>> 'source_urls': [('http://sourceforge.net/projects/numpy/files/NumPy/%s' % >>>> numpyversion, 'download')], >>>> 'patches': [ >>>> 'numpy-%s_distutils_multiple-lib-dirs.patch' % numpyversion, >>>> 'numpy-iqacml.patch', >>>> ], >>>> }), >>>> ] >>>> >>>> moduleclass = 'lang' >>>> >>>> >>>> >>>> With debugging turned on (EASYBUILD_DEBUG=1), I get an error building the >>>> module numpy. >>>> >>>> == 2013-12-18 14:39:49,745 main.filetools WARNING Found 5 errors in >>>> command output (output: collect2: error: ld returned 1 exit status, >>>> collect2: error: ld returned 1 exit status, collect2: error: ld returned 1 >>>> exit status, collect2: error: ld returned 1 exit status, collect2: error: >>>> ld returned 1 exit status) >>>> == 2013-12-18 14:39:49,745 main.environment INFO Environment variable >>>> PYTHONPATH set to >>>> /hpc-common/software/easybuild/inl-easyconfigs:/hpc-common/software/easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages >>>> == 2013-12-18 14:39:50,759 main.filetools INFO Path >>>> /apps/local/easybuild/build/Python/2.7.5/gnupack-1.1.1/numpy successfully >>>> removed. >>>> == 2013-12-18 14:39:50,761 main.Lmod INFO $MODULEPATH set based on list of >>>> module paths: >>>> /tmp/tmpT3IMq2/all:/apps/local/easybuild/modules/all:/apps/local/modules/modulefiles:/apps/projects/modules/modulefiles:/apps/local/modules/modulefiles/Linux:/apps/local/modules/modulefiles/Core >>>> == 2013-12-18 14:39:50,762 main.filetools INFO Command lmod found at >>>> /apps/projects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod >>>> == 2013-12-18 14:39:50,762 main.Lmod INFO Full path for module command is >>>> /apps/projects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod, so using it >>>> == 2013-12-18 14:39:50,763 main.Lmod ERROR EasyBuild crashed with an error >>>> (at >>>> easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages/easybuild/tools/modules.py:625 >>>> in __init__): Failed to check Lmod version: [Errno 2] No such file or >>>> directory >>>> >>>> The exception is occurring at the line below (file modules.py, function >>>> run_module), when it tries to execute os.getcwd()? >>>> >>>> self.log.debug("Running module command '%s' from %s" % (' '.join(cmdlist + >>>> args), os.getcwd())) >>>> >>>> >>>> When I turn debugging-off >>>> >>>> I get the following error >>>> >>>> == 2013-12-18 16:13:28,127 main.filetools WARNING Found 5 errors in >>>> command output (output: collect2: error: ld returned 1 exit status, >>>> collect2: error: ld returned 1 exit status, collect2: error: ld returned 1 >>>> exit status, collect2: error: ld returned 1 exit status, collect2: error: >>>> ld returned 1 exit status) >>>> == 2013-12-18 16:13:28,128 main.environment INFO Environment variable >>>> PYTHONPATH set to >>>> /hpc-common/software/easybuild/inl-easyconfigs:/hpc-common/software/easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages >>>> == 2013-12-18 16:13:29,178 main.filetools INFO Path >>>> /apps/local/easybuild/build/Python/2.7.5/gnupack-1.1.1/numpy successfully >>>> removed. >>>> == 2013-12-18 16:13:29,181 main.Lmod INFO $MODULEPATH set based on list of >>>> module paths: >>>> /tmp/tmpv_sTqF/all:/apps/local/easybuild/modules/all:/apps/local/modules/modulefiles:/apps/projects/modules/modulefiles:/apps/local/modules/modulefiles/Linux:/apps/local/modules/modulefiles/Core >>>> == 2013-12-18 16:13:29,182 main.filetools INFO Command lmod found at >>>> /apps/projects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod >>>> == 2013-12-18 16:13:29,182 main.Lmod INFO Full path for module command is >>>> /apps/projects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod, so using it >>>> == 2013-12-18 16:13:29,182 main.Lmod INFO Before module update >>>> == 2013-12-18 16:13:29,269 main.Lmod INFO After module update >>>> == 2013-12-18 16:13:29,270 main.Lmod INFO After tempfile >>>> == 2013-12-18 16:13:29,270 main.Lmod INFO Before open >>>> == 2013-12-18 16:13:29,270 main.Lmod INFO After open >>>> == 2013-12-18 16:13:29,355 main.Lmod INFO After subprocess >>>> == 2013-12-18 16:13:29,356 main.Lmod INFO After open >>>> == 2013-12-18 16:13:29,356 main.Lmod INFO After read >>>> == 2013-12-18 16:13:29,356 main.Lmod INFO lmod help output: >>>> /apps/projects/lua/5.1.4.5/gcc-4.5.1/opt/bin/lua: >>>> ...gcc-4.5.1/opt/lmod/lmod/libexec/../tools/fileOps.lua:295: bad argument >>>> #1 to 'chdir' (string expected, got nil) >>>> stack traceback: >>>> [C]: in function 'chdir' >>>> ...gcc-4.5.1/opt/lmod/lmod/libexec/../tools/fileOps.lua:295: in function >>>> 'abspath' >>>> ...s/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/utils.lua:427: in function >>>> 'readRC' >>>> ...ojects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod:472: in function >>>> 'main' >>>> ...ojects/lmod/5.2/gcc-4.5.1/opt/lmod/lmod/libexec/lmod:618: in main chunk >>>> [C]: ? >>>> >>>> >>>> Any suggestions? >>>> >>>> Thanks, >>>> Cormac. >>>> >>>> >>>> >>>> -- >>>> Cormac Garvey >>>> HPC Software Consultant >>>> Scientific Computing >>>> Idaho National Laboratory >>>> Ph: 208-526-6294 >>>> >>>> >>>> >>>> >>>> -- >>>> Cormac Garvey >>>> HPC Software Consultant >>>> Scientific Computing >>>> Idaho National Laboratory >>>> Ph: 208-526-6294 >>>> >>> >>> >>> >>> >>> -- >>> Cormac Garvey >>> HPC Software Consultant >>> Scientific Computing >>> Idaho National Laboratory >>> Ph: 208-526-6294 >>> >> >> >> >> >> -- >> Cormac Garvey >> HPC Software Consultant >> Scientific Computing >> Idaho National Laboratory >> Ph: 208-526-6294 >> >> <easybuild-rsUBi8.log.gz> >
