Re: [Numpy-discussion] Unable to building numpy with openblas using bento or distutils

2013-03-23 Thread Hanno Klemm 
Skipper,
this looks like a problem that I had in the bad old days with ATLAS, as well. 
Try compiling openblas with the -fPIC flag that used to help. 

Best of luck,
Hanno

hanno.kl...@me.com

Sent from my mobile device, please excuse my brevity. 

On 23.03.2013, at 19:19, Skipper Seabold jsseab...@gmail.com wrote:

 Some help on this would be greatly appreciated. It's been recommended to use 
 OpenBlas over ATLAS, so I've been trying to build numpy with openblas and 
 have run into a few problems.
 
 1) Build fails using bento master and waf 1.7.9, see below.
 2) Distutils doesn't seem to be able to find lapack as part of atlas. I tried 
 to skip a site.cfg and define environmental variables. No idea what I missed.
 
 I followed instructions found scattered over the internet and only understand 
 vaguely the issues. Maybe someone can help. I'd be happy to update the wiki 
 with any answers. 
 
 To truly support OpenBlas, is it maybe necessary to make some additions to 
 numpy/distutils/system_info.py?
 
 Thanks for having a look,
 
 Skipper
 
 Install OpenBlas
 -
 git clone git://github.com/xianyi/OpenBLAS
 cd OpenBlas
 
 Edit c_check to look for libpthreads in the right place (Kubuntu 12.10)
 
 |4 $ git diff c_check
 ```
 diff --git a/c_check b/c_check
 index 4d82237..de0fd33 100644
 --- a/c_check
 +++ b/c_check
 @@ -241,7 +241,7 @@ print CONFFILE #define FUNDERSCORE\t$need_fu\n if 
 $need_fu
  
  if ($os eq LINUX) {
  
 -@pthread = split(/\s+/, `nm /lib/libpthread.so* | grep _pthread_create`);
 +@pthread = split(/\s+/, `nm /lib/x86_64-linux-gnu/libpthread.so* | grep 
 _pthread_create`);
  
  if ($pthread[2] ne ) {
 print CONFFILE #define PTHREAD_CREATE_FUNC $pthread[2]\n;
 ```
 
 make fc=gfortran
 make PREFIX=~/.local install
 
 Everything looks ok, so far. 
 
 Install NumPy
 ---
 Using numpy master
 
 I tried to use bento master and waf 1.7.9, following instructions from 
 David's blog
 
 bentomaker configure --prefix=/home/skipper/.local 
 --with-blas-lapack-libdir=/home/skipper/.local/lib 
 --blas-lapack-type=openblas ..
 bentomaker build -j4
 
 ```
 snip
 [101/104] cshlib: build/numpy/core/src/umath/umath_tests.c.11.o - 
 build/numpy/core/umath_tests.so   

 /usr/bin/ld: numpy/core/lib/libnpymath.a(halffloat.c.12.o): relocation 
 R_X86_64_PC32 against symbol `npy_halfbits_to_floatbits' can not be used when 
 making a shared object; recompile with -fPIC
 /usr/bin/ld: final link failed: Bad value
 collect2: error: ld returned 1 exit status
 /usr/bin/ld: numpy/core/lib/libnpymath.a(halffloat.c.12.o): relocation 
 R_X86_64_PC32 against symbol `npy_halfbits_to_floatbits' can not be used when 
 making a shared object; recompile with -fPIC
 /usr/bin/ld: final link failed: Bad value
 collect2: error: ld returned 1 exit status
 ```
 
 No idea, so, let's try distutils
 
 export LAPACK=~/.local/lib/libopenblas.a
 export BLAS=~/.local/lib/libopenblas.a
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.local/lib/
 echo $LD_LIBRARY_PATH
 ```
 :/usr/local/lib64/R/bin:/home/skipper/.local/lib/
 ```
 
 This step seems to be necessary?
 
 python setup.py config
 ```
 Running from numpy source directory.
 non-existing path in 'numpy/distutils': 'site.cfg'
 F2PY Version 2
 numpy/core/setup_common.py:88: MismatchCAPIWarning: API mismatch detected, 
 the C API version numbers have to be updated. Current C api version is 8, 
 with checksum f4362353e2d72f889fda0128aa015037, but recorded checksum for C 
 API version 8 in codegen_dir/cversions.txt is 
 17321775fc884de0b1eda478cd61c74b. If functions were added in the C API, you 
 have to update C_API_VERSION  in numpy/core/setup_common.py.
   MismatchCAPIWarning)
 blas_opt_info:
 blas_mkl_info:
   libraries mkl,vml,guide not found in ['/usr/local/lib64', '/usr/local/lib', 
 '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
   NOT AVAILABLE
 
 atlas_blas_threads_info:
 Setting PTATLAS=ATLAS
   libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib64', 
 '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
   NOT AVAILABLE
 
 atlas_blas_info:
   libraries f77blas,cblas,atlas not found in ['/usr/local/lib64', 
 '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
   NOT AVAILABLE
 
 /home/skipper/src/numpy-skipper/numpy/distutils/system_info.py:1501: 
 UserWarning: 
 Atlas (http://math-atlas.sourceforge.net/) libraries not found.
 Directories to search for the libraries can be specified in the
 numpy/distutils/site.cfg file (section [atlas]) or by setting
 the ATLAS environment variable.
   warnings.warn(AtlasNotFoundError.__doc__)
 blas_info:
 Replacing _lib_names[0]=='blas' with 'openblas'
 Replacing _lib_names[0]=='openblas' with 'openblas'
   FOUND:
 libraries = ['openblas']
 library_dirs = ['/home/skipper/.local/lib']
 language = f77
 
   FOUND:

Re: [Numpy-discussion] Unable to building numpy with openblas using bento or distutils

2013-03-23 Thread Skipper Seabold 
On Sat, Mar 23, 2013 at 2:32 PM, Hanno Klemm kl...@phys.ethz.ch wrote:

 Skipper,
 this looks like a problem that I had in the bad old days with ATLAS, as
 well. Try compiling openblas with the -fPIC flag that used to help.


Thanks for having a look. I checked after seeing that odd bento failure
(see here [1]), and it looks to me like OpenBlas uses the -fPIC flag in all
of the gcc and gfortran calls. Possible related? [2]

Skipper

[1] https://github.com/cournape/Bento/issues/116
[2] https://github.com/cournape/Bento/issues/128



 Best of luck,
 Hanno

 hanno.kl...@me.com

 Sent from my mobile device, please excuse my brevity.

 On 23.03.2013, at 19:19, Skipper Seabold jsseab...@gmail.com wrote:

 Some help on this would be greatly appreciated. It's been recommended to
 use OpenBlas over ATLAS, so I've been trying to build numpy with openblas
 and have run into a few problems.

 1) Build fails using bento master and waf 1.7.9, see below.
 2) Distutils doesn't seem to be able to find lapack as part of atlas. I
 tried to skip a site.cfg and define environmental variables. No idea what I
 missed.

 I followed instructions found scattered over the internet and only
 understand vaguely the issues. Maybe someone can help. I'd be happy to
 update the wiki with any answers.

 To truly support OpenBlas, is it maybe necessary to make some additions to
 numpy/distutils/system_info.py?

 Thanks for having a look,

 Skipper

 Install OpenBlas
 -
 git clone git://github.com/xianyi/OpenBLAS
 cd OpenBlas

 Edit c_check to look for libpthreads in the right place (Kubuntu 12.10)

 |4 $ git diff c_check
 ```
 diff --git a/c_check b/c_check
 index 4d82237..de0fd33 100644
 --- a/c_check
 +++ b/c_check
 @@ -241,7 +241,7 @@ print CONFFILE #define FUNDERSCORE\t$need_fu\n if
 $need_fu

  if ($os eq LINUX) {

 -@pthread = split(/\s+/, `nm /lib/libpthread.so* | grep
 _pthread_create`);
 +@pthread = split(/\s+/, `nm /lib/x86_64-linux-gnu/libpthread.so* |
 grep _pthread_create`);

  if ($pthread[2] ne ) {
 print CONFFILE #define PTHREAD_CREATE_FUNC $pthread[2]\n;
 ```

 make fc=gfortran
 make PREFIX=~/.local install

 Everything looks ok, so far.

 Install NumPy
 ---
 Using numpy master

 I tried to use bento master and waf 1.7.9, following instructions from
 David's blog

 bentomaker configure --prefix=/home/skipper/.local
 --with-blas-lapack-libdir=/home/skipper/.local/lib
 --blas-lapack-type=openblas ..
 bentomaker build -j4

 ```
 snip
 [101/104] cshlib: build/numpy/core/src/umath/umath_tests.c.11.o -
 build/numpy/core/umath_tests.so

 /usr/bin/ld: numpy/core/lib/libnpymath.a(halffloat.c.12.o): relocation
 R_X86_64_PC32 against symbol `npy_halfbits_to_floatbits' can not be used
 when making a shared object; recompile with -fPIC
 /usr/bin/ld: final link failed: Bad value
 collect2: error: ld returned 1 exit status
 /usr/bin/ld: numpy/core/lib/libnpymath.a(halffloat.c.12.o): relocation
 R_X86_64_PC32 against symbol `npy_halfbits_to_floatbits' can not be used
 when making a shared object; recompile with -fPIC
 /usr/bin/ld: final link failed: Bad value
 collect2: error: ld returned 1 exit status
 ```

 No idea, so, let's try distutils

 export LAPACK=~/.local/lib/libopenblas.a
 export BLAS=~/.local/lib/libopenblas.a
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.local/lib/
 echo $LD_LIBRARY_PATH
 ```
 :/usr/local/lib64/R/bin:/home/skipper/.local/lib/
 ```

 This step seems to be necessary?

 python setup.py config
 ```
 Running from numpy source directory.
 non-existing path in 'numpy/distutils': 'site.cfg'
 F2PY Version 2
 numpy/core/setup_common.py:88: MismatchCAPIWarning: API mismatch detected,
 the C API version numbers have to be updated. Current C api version is 8,
 with checksum f4362353e2d72f889fda0128aa015037, but recorded checksum for C
 API version 8 in codegen_dir/cversions.txt is
 17321775fc884de0b1eda478cd61c74b. If functions were added in the C API, you
 have to update C_API_VERSION  in numpy/core/setup_common.py.
   MismatchCAPIWarning)
 blas_opt_info:
 blas_mkl_info:
   libraries mkl,vml,guide not found in ['/usr/local/lib64',
 '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
   NOT AVAILABLE

 atlas_blas_threads_info:
 Setting PTATLAS=ATLAS
   libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib64',
 '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
   NOT AVAILABLE

 atlas_blas_info:
   libraries f77blas,cblas,atlas not found in ['/usr/local/lib64',
 '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
   NOT AVAILABLE

 /home/skipper/src/numpy-skipper/numpy/distutils/system_info.py:1501:
 UserWarning:
 Atlas (http://math-atlas.sourceforge.net/) libraries not found.
 Directories to search for the libraries can be specified in the
 numpy/distutils/site.cfg file (section [atlas]) or by setting
 the ATLAS environment variable.

Re: [Numpy-discussion] Unable to building numpy with openblas using bento or distutils

2013-03-23 Thread Ake Sandgren 
On Sat, 2013-03-23 at 14:19 -0400, Skipper Seabold wrote:
 Some help on this would be greatly appreciated. It's been recommended
 to use OpenBlas over ATLAS, so I've been trying to build numpy with
 openblas and have run into a few problems.

 
 To truly support OpenBlas, is it maybe necessary to make some
 additions to numpy/distutils/system_info.py?

Here is how.

https://github.com/akesandgren/numpy/commit/363339dd3a9826f3e3e7dc4248c258d3c4dfcd7c

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Unable to building numpy with openblas using bento or distutils

2013-03-23 Thread Skipper Seabold 
On Sat, Mar 23, 2013 at 7:26 PM, Ake Sandgren ake.sandg...@hpc2n.umu.se wrote:
 On Sat, 2013-03-23 at 14:19 -0400, Skipper Seabold wrote:
 Some help on this would be greatly appreciated. It's been recommended
 to use OpenBlas over ATLAS, so I've been trying to build numpy with
 openblas and have run into a few problems.


 To truly support OpenBlas, is it maybe necessary to make some
 additions to numpy/distutils/system_info.py?

 Here is how.

 https://github.com/akesandgren/numpy/commit/363339dd3a9826f3e3e7dc4248c258d3c4dfcd7c



Thanks that works well for numpy. Test pass. I hope that makes it into
a pull request. My site.cfg looks like this. I don't know about the
lapack_opt section. It doesn't seem to work.

[DEFAULT]
library_dirs = /home/skipper/.local/lib
include_dirs = /home/skipper/.local/include

[openblas]
libraries = openblas

[lapack_opt]
libraries = openblas

Do you have any idea how to get scipy working too. I have a similar
site.cfg, but it does not find lapack, which is rolled into
libopenblas from what I understand. I can do

export LAPACK=~/.local/lib/libopenblas.a
python setup.py build  build.log
sudo -E python setup.py install

There are no obvious failures in the build.log, but scipy is still
broken because it needs lapack from numpy I guess.

 import numpy as np
 np.show_config()
lapack_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
libraries = ['openblas', 'openblas']
library_dirs = ['/home/skipper/.local/lib']
language = f77
lapack_src_info:
  NOT AVAILABLE
openblas_info:
libraries = ['openblas', 'openblas']
library_dirs = ['/home/skipper/.local/lib']
language = f77
lapack_opt_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE
 from scipy import stats
Traceback (most recent call last):
  File stdin, line 1, in module
  File /usr/local/lib/python2.7/dist-packages/scipy/stats/__init__.py,
line 320, in module
from .stats import *
  File /usr/local/lib/python2.7/dist-packages/scipy/stats/stats.py,
line 242, in module
import scipy.linalg as linalg
  File /usr/local/lib/python2.7/dist-packages/scipy/linalg/__init__.py,
line 147, in module
from .misc import *
  File /usr/local/lib/python2.7/dist-packages/scipy/linalg/misc.py,
line 5, in module
from . import blas
  File /usr/local/lib/python2.7/dist-packages/scipy/linalg/blas.py,
line 113, in module
from scipy.linalg import _fblas
ImportError: libopenblas.so.0: cannot open shared object file: No such
file or directory

Skipper
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Unable to building numpy with openblas using bento or distutils

2013-03-23 Thread Skipper Seabold 
On Sat, Mar 23, 2013 at 8:44 PM, Skipper Seabold jsseab...@gmail.com wrote:
 On Sat, Mar 23, 2013 at 7:26 PM, Ake Sandgren ake.sandg...@hpc2n.umu.se 
 wrote:
 On Sat, 2013-03-23 at 14:19 -0400, Skipper Seabold wrote:
 Some help on this would be greatly appreciated. It's been recommended
 to use OpenBlas over ATLAS, so I've been trying to build numpy with
 openblas and have run into a few problems.


 To truly support OpenBlas, is it maybe necessary to make some
 additions to numpy/distutils/system_info.py?

 Here is how.

 https://github.com/akesandgren/numpy/commit/363339dd3a9826f3e3e7dc4248c258d3c4dfcd7c



 Thanks that works well for numpy. Test pass. I hope that makes it into
 a pull request. My site.cfg looks like this. I don't know about the
 lapack_opt section. It doesn't seem to work.

 [DEFAULT]
 library_dirs = /home/skipper/.local/lib
 include_dirs = /home/skipper/.local/include

 [openblas]
 libraries = openblas

 [lapack_opt]
 libraries = openblas

 Do you have any idea how to get scipy working too. I have a similar
 site.cfg, but it does not find lapack, which is rolled into
 libopenblas from what I understand. I can do

 export LAPACK=~/.local/lib/libopenblas.a
 python setup.py build  build.log
 sudo -E python setup.py install

 There are no obvious failures in the build.log, but scipy is still
 broken because it needs lapack from numpy I guess.

The answer is to

export BLAS=~/.local/lib/libopenblas.a
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.local/lib/

before building and installing. Now everything works. Whew. Thanks a
lot for the help.


 import numpy as np
 np.show_config()
 lapack_info:
   NOT AVAILABLE
 atlas_threads_info:
   NOT AVAILABLE
 blas_opt_info:
 libraries = ['openblas', 'openblas']
 library_dirs = ['/home/skipper/.local/lib']
 language = f77
 lapack_src_info:
   NOT AVAILABLE
 openblas_info:
 libraries = ['openblas', 'openblas']
 library_dirs = ['/home/skipper/.local/lib']
 language = f77
 lapack_opt_info:
   NOT AVAILABLE
 atlas_info:
   NOT AVAILABLE
 lapack_mkl_info:
   NOT AVAILABLE
 blas_mkl_info:
   NOT AVAILABLE
 mkl_info:
   NOT AVAILABLE
 from scipy import stats
 Traceback (most recent call last):
   File stdin, line 1, in module
   File /usr/local/lib/python2.7/dist-packages/scipy/stats/__init__.py,
 line 320, in module
 from .stats import *
   File /usr/local/lib/python2.7/dist-packages/scipy/stats/stats.py,
 line 242, in module
 import scipy.linalg as linalg
   File /usr/local/lib/python2.7/dist-packages/scipy/linalg/__init__.py,
 line 147, in module
 from .misc import *
   File /usr/local/lib/python2.7/dist-packages/scipy/linalg/misc.py,
 line 5, in module
 from . import blas
   File /usr/local/lib/python2.7/dist-packages/scipy/linalg/blas.py,
 line 113, in module
 from scipy.linalg import _fblas
 ImportError: libopenblas.so.0: cannot open shared object file: No such
 file or directory

 Skipper
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion