subject:"\[easybuild\] Problem with include and lib paths for python modules"

Re: [easybuild] Problem with include and lib paths for python modules

2016-11-02 Thread Vanzo, Davide

Hi Kenneth,
I will look into Quassel, although it still seem that Quassel Core still needs 
to be hosted on a system that is always connected too. I will have to arrange 
something... Anyway, thanks for the suggestion.

Back to our problem. For numpy I used a modified version of 
numpy-1.8.2-foss-2016a-Python-2.7.11.eb where the only changes were in the 
toolchain (goolf instead of foss) and Python version. Attached you can find the 
easyconfig and the resulting lua module. The path to the numpy include 
directory does not appear on CPATH once loading the module.

$ ml goolf
$ ml numpy
$ ml

Currently Loaded Modules:
 1) GCCcore/.5.4.0   4) numactl/.2.0.11   7) OpenBLAS/0.2.19-LAPACK-3.6.1   
   10) goolf/2016.10  13) ncurses/.6.0  16) SQLite/3.15.0  19) 
libffi/.3.2.122) numpy/1.11.2-Python-2.7.12
 2) binutils/.2.26   5) hwloc/.1.11.4 8) FFTW/3.3.5 
   11) bzip2/.1.0.6   14) libreadline/.7.0  17) Tk/8.6.6   20) 
OpenSSL/.1.0.2j
 3) GCC/5.4.0-2.26   6) OpenMPI/1.10.49) 
ScaLAPACK/2.0.2-OpenBLAS-0.2.19-LAPACK-3.6.1  12) zlib/.1.2.815) Tcl/8.6.6  
   18) GMP/.6.1.1 21) Python/2.7.12

$ echo $CPATH | tr ':' '\n'
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/OpenSSL/1.0.2j/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/GMP/6.1.1/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Tk/8.6.6/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/SQLite/3.15.0/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Tcl/8.6.6/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/libreadline/7.0/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/ncurses/6.0/include
/opt/easybuild/software/Compiler/GCCcore/5.4.0/zlib/1.2.8/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/bzip2/1.0.6/include
/opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/ScaLAPACK/2.0.2-OpenBLAS-0.2.19-LAPACK-3.6.1/include
/opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/FFTW/3.3.5/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/OpenBLAS/0.2.19-LAPACK-3.6.1/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/OpenMPI/1.10.4/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/hwloc/1.11.4/include
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/numactl/2.0.11/include
/opt/easybuild/software/Compiler/GCCcore/5.4.0/binutils/2.26/include
/opt/easybuild/software/Core/GCCcore/5.4.0/include

--
Davide Vanzo, PhD
Application Developer
Adjunct Assistant Professor of Chemical and Biomolecular Engineering
Advanced Computing Center for Research and Education (ACCRE)
Vanderbilt University - Hill Center 201
(615)-875-9137
www.accre.vanderbilt.edu

On Nov 2 2016, at 8:28 am, Kenneth Hoste  wrote:

Hi Davide,

> On 1 Nov 2016, at 18:07, Vanzo, Davide  wrote:
>
> Hello,
> I am moving this discussion over here since it is hard to follow up on the 
> IRC channel (I don't have a computer online 24/7 to set up a bouncer and the 
> time zone differential from Europe does not help either).

You could use Quassel, then you have a 'server' component that is always 
online, and you use a client to connect to it (talk to Pablo on the #easybuild 
IRC channel, he can hook you up). That doesn't fix the timezone issue though. 
;-)

So, it makes sense to move the discussion here.

> I am trying to build ScientificPython and it fails with the following error:
>
> Include/Scientific/arrayobject.h:2:30: fatal error: numpy/oldnumeric.h: No 
> such file or directory
>
> The reason can be found in the fact that in toolchain.py it looks for 
> /include, /lib and /lib64 to obtain the paths 
> that need to be passed to the compiler.

That's not entirely accurate... The toolchain mechanism does define the build 
environment ($CC, $CFLAGS, ...), but $CPATH (which specifies the locations that 
the compiler should consider for header files) is defined in the generated 
module files.

> Although those are the default paths for the majority of programs, python 
> modules have a different path hierarchy and it fails because for numpy the 
> headers are in /lib/python2.7/site-packages/numpy/core/include. See 
> the following debug log extract for evidence.
>
> So, how would you suggest to proceed? One way would be to have a recursive 
> search starting from  coupled with a more tight check on the 
> lib/include paths based not only on the directory name but also on its 
> content (are there .a, .o or .h files?). That would also solve the problem of 
> false positive entries like /lib that does not contain library 
> files at all.

I'm actually quite surprised this only pops up now... Doesn't the module file 
for numpy correctly define $CPATH? (can't easily check myself atm)

Which easyconfig file are you using exactly? Maybe this is just an issue with a 
too recent numpy version being used? I seem to recall that ScientificPython 
requires

Re: [easybuild] Problem with include and lib paths for python modules

2016-11-02 Thread Kenneth Hoste

Hi Davide,

> On 1 Nov 2016, at 18:07, Vanzo, Davide  wrote:
> 
> Hello,
> I am moving this discussion over here since it is hard to follow up on the 
> IRC channel (I don't have a computer online 24/7 to set up a bouncer and the 
> time zone differential from Europe does not help either).

You could use Quassel, then you have a 'server' component that is always 
online, and you use a client to connect to it (talk to Pablo on the #easybuild 
IRC channel, he can hook you up). That doesn't fix the timezone issue though. 
;-)

So, it makes sense to move the discussion here.

> I am trying to build ScientificPython and it fails with the following error:
> 
>   Include/Scientific/arrayobject.h:2:30: fatal error: 
> numpy/oldnumeric.h: No such file or directory
> 
> The reason can be found in the fact that in toolchain.py it looks for 
> /include, /lib and /lib64 to obtain the paths 
> that need to be passed to the compiler.

That's not entirely accurate... The toolchain mechanism does define the build 
environment ($CC, $CFLAGS, ...), but $CPATH (which specifies the locations that 
the compiler should consider for header files) is defined in the generated 
module files.

> Although those are the default paths for the majority of programs, python 
> modules have a different path hierarchy and it fails because for numpy the 
> headers are in /lib/python2.7/site-packages/numpy/core/include. See 
> the following debug log extract for evidence.
> 
> So, how would you suggest to proceed? One way would be to have a recursive 
> search starting from  coupled with a more tight check on the 
> lib/include paths based not only on the directory name but also on its 
> content (are there .a, .o or .h files?). That would also solve the problem of 
> false positive entries like /lib that does not contain library 
> files at all.

I'm actually quite surprised this only pops up now... Doesn't the module file 
for numpy correctly define $CPATH? (can't easily check myself atm)

Which easyconfig file are you using exactly? Maybe this is just an issue with a 
too recent numpy version being used? I seem to recall that ScientificPython 
requires an old version of numpy...


regards,

Kenneth

> 
> Davide
> 
> 
> 
> == 2016-11-01 12:05:51,970 toolchain.py:218 DEBUG set_variables: toolchain 
> variables. Do nothing.
> == 2016-11-01 12:05:51,970 toolchain.py:647 DEBUG prepare: set additional 
> variables onlymod=False
> == 2016-11-01 12:05:51,970 toolchain.py:270 DEBUG get_software_root software 
> root /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12 for Python 
> was found in environment
> == 2016-11-01 12:05:51,971 toolchain.py:270 DEBUG get_software_root software 
> root 
> /opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12
>  for numpy was found in environment
> == 2016-11-01 12:05:51,971 variables.py:542 DEBUG Passthrough to LISTCLASS 
> element function append_subdirs
> == 2016-11-01 12:05:51,971 variables.py:267 DEBUG _is_protected: False value 
> None (None)
> == 2016-11-01 12:05:51,971 variables.py:297 DEBUG nappend: value None 
> newvalue [] position None
> == 2016-11-01 12:05:51,971 variables.py:186 DEBUG append_subdirs: base 
> /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12 subdirs 
> ['include']
> == 2016-11-01 12:05:51,972 variables.py:198 DEBUG append_subdirs: added 
> directory 
> /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12/include
> == 2016-11-01 12:05:51,972 variables.py:542 DEBUG Passthrough to LISTCLASS 
> element function append_subdirs
> == 2016-11-01 12:05:51,972 variables.py:267 DEBUG _is_protected: False value 
> None (None)
> == 2016-11-01 12:05:51,972 variables.py:297 DEBUG nappend: value None 
> newvalue [] position None
> == 2016-11-01 12:05:51,972 variables.py:186 DEBUG append_subdirs: base 
> /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12 subdirs 
> ['lib64', 'lib']
> == 2016-11-01 12:05:51,973 variables.py:200 WARNING flags_for_subdirs: 
> directory /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12/lib64 
> was not found
> == 2016-11-01 12:05:51,973 variables.py:198 DEBUG append_subdirs: added 
> directory /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12/lib
> == 2016-11-01 12:05:51,973 variables.py:542 DEBUG Passthrough to LISTCLASS 
> element function append_subdirs
> == 2016-11-01 12:05:51,973 variables.py:267 DEBUG _is_protected: False value 
> None (None)
> == 2016-11-01 12:05:51,973 variables.py:297 DEBUG nappend: value None 
> newvalue [] position None
> == 2016-11-01 12:05:51,974 variables.py:186 DEBUG append_subdirs: base 
> /opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12
>  subdirs ['include']
> == 2016-11-01 12:05:51,974 variables.py:200 WARNING flags_for_subdirs: 
> directory 
> /opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12/include
>  was not found
> == 2016-11-01

[easybuild] Problem with include and lib paths for python modules

2016-11-01 Thread Vanzo, Davide

Hello,
I am moving this discussion over here since it is hard to follow up on the IRC 
channel (I don't have a computer online 24/7 to set up a bouncer and the time 
zone differential from Europe does not help either).

I am trying to build ScientificPython and it fails with the following error:

  Include/Scientific/arrayobject.h:2:30: fatal error: 
numpy/oldnumeric.h: No such file or directory

The reason can be found in the fact that in toolchain.py it looks for 
/include, /lib and /lib64 to obtain the paths 
that need to be passed to the compiler. Although those are the default paths 
for the majority of programs, python modules have a different path hierarchy 
and it fails because for numpy the headers are in 
/lib/python2.7/site-packages/numpy/core/include. See the following 
debug log extract for evidence.

So, how would you suggest to proceed? One way would be to have a recursive 
search starting from  coupled with a more tight check on the 
lib/include paths based not only on the directory name but also on its content 
(are there .a, .o or .h files?). That would also solve the problem of false 
positive entries like /lib that does not contain library files at 
all.

Davide



== 2016-11-01 12:05:51,970 toolchain.py:218 DEBUG set_variables: toolchain 
variables. Do nothing.
== 2016-11-01 12:05:51,970 toolchain.py:647 DEBUG prepare: set additional 
variables onlymod=False
== 2016-11-01 12:05:51,970 toolchain.py:270 DEBUG get_software_root software 
root /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12 for Python 
was found in environment
== 2016-11-01 12:05:51,971 toolchain.py:270 DEBUG get_software_root software 
root 
/opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12
 for numpy was found in environment
== 2016-11-01 12:05:51,971 variables.py:542 DEBUG Passthrough to LISTCLASS 
element function append_subdirs
== 2016-11-01 12:05:51,971 variables.py:267 DEBUG _is_protected: False value 
None (None)
== 2016-11-01 12:05:51,971 variables.py:297 DEBUG nappend: value None newvalue 
[] position None
== 2016-11-01 12:05:51,971 variables.py:186 DEBUG append_subdirs: base 
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12 subdirs 
['include']
== 2016-11-01 12:05:51,972 variables.py:198 DEBUG append_subdirs: added 
directory /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12/include
== 2016-11-01 12:05:51,972 variables.py:542 DEBUG Passthrough to LISTCLASS 
element function append_subdirs
== 2016-11-01 12:05:51,972 variables.py:267 DEBUG _is_protected: False value 
None (None)
== 2016-11-01 12:05:51,972 variables.py:297 DEBUG nappend: value None newvalue 
[] position None
== 2016-11-01 12:05:51,972 variables.py:186 DEBUG append_subdirs: base 
/opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12 subdirs ['lib64', 
'lib']
== 2016-11-01 12:05:51,973 variables.py:200 WARNING flags_for_subdirs: 
directory /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12/lib64 
was not found
== 2016-11-01 12:05:51,973 variables.py:198 DEBUG append_subdirs: added 
directory /opt/easybuild/software/Compiler/GCC/5.4.0-2.26/Python/2.7.12/lib
== 2016-11-01 12:05:51,973 variables.py:542 DEBUG Passthrough to LISTCLASS 
element function append_subdirs
== 2016-11-01 12:05:51,973 variables.py:267 DEBUG _is_protected: False value 
None (None)
== 2016-11-01 12:05:51,973 variables.py:297 DEBUG nappend: value None newvalue 
[] position None
== 2016-11-01 12:05:51,974 variables.py:186 DEBUG append_subdirs: base 
/opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12
 subdirs ['include']
== 2016-11-01 12:05:51,974 variables.py:200 WARNING flags_for_subdirs: 
directory 
/opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12/include
 was not found
== 2016-11-01 12:05:51,974 variables.py:542 DEBUG Passthrough to LISTCLASS 
element function append_subdirs
== 2016-11-01 12:05:51,974 variables.py:267 DEBUG _is_protected: False value 
None (None)
== 2016-11-01 12:05:51,974 variables.py:297 DEBUG nappend: value None newvalue 
[] position None
== 2016-11-01 12:05:51,975 variables.py:186 DEBUG append_subdirs: base 
/opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12
 subdirs ['lib64', 'lib']
== 2016-11-01 12:05:51,975 variables.py:200 WARNING flags_for_subdirs: 
directory 
/opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12/lib64
 was not found
== 2016-11-01 12:05:51,975 variables.py:198 DEBUG append_subdirs: added 
directory 
/opt/easybuild/software/MPI/GCC/5.4.0-2.26/OpenMPI/1.10.4/numpy/1.8.2-Python-2.7.12/lib

Re: [easybuild] Problem with include and lib paths for python modules

Re: [easybuild] Problem with include and lib paths for python modules

[easybuild] Problem with include and lib paths for python modules

3 matches

Site Navigation

Mail list logo

Footer information