Re: [OE-core] python recipes with extensions fail to find/link with proper libraries

[Francesco Giancane]

Hi Alejandro,

Thank you for your support. I will try to explain better the situation. I
can attach the recipes but it is useless, the problem is not at recipe
level.

python3-numpy has an optional dependency on BLAS interface, and openblas is
an implementation of that interface.
python3-scipy has a mandatory dependency on BLAS interface, and openblas is
an implementation of that interface.

So, as simple as it seems, I added a python3-numpy_%.bbappend with
DEPENDS += "openblas".

Given the fact that openblas has a lot of issues when cross compiling, I
compiled it separately on the target (Raspberry Pi 3) and then the built
package has been tar'ed and gzip'ed.

After this step, I wrote a simple recipe (openblas.bb) which install as
binary files the content of the aforementioned tar archive (obviously
setting the compatible machine with arm only). At the end of the day, this
is just a not-so-clever hack to have a working openblas on the target,
including -dev packages.

After all this effort, I expected the job to be done: given that, after my
append file, the python3-numpy recipe depends on openblas, it will be
scheduled only after openblas.
At that point, given that the recipe for python3-numpy has access to the
include files and the library development files, I expect that running
setup.py would suffice in order to find & link the python C extensions to
openblas.
Unfortunately, given that the setup.py script is executed by the
python3-native package, the path the interpreter is searching for the
native libraries are not the ones for the sys root... And thus when
setup.py searches for libopenblas.so(arm) it will fail as openblas is not
deployed on x86_64 (native) variant.

Sorry for the very long mail, but I hope that the scenario is now clearer.

Long story short, python setuptools is failing to find correct libraries
for the target, and is searching for them in the native path, where it
won't find them. (Look at my first email with log attached).

Thanks,
Francesco Giancane


Il giorno lun 7 mag 2018 alle ore 23:28 Alejandro Enedino Hernandez
Samaniego  ha scritto:

> Hello Francesco,
>
>
> Please hit the reply-all button, otherwise I wont be getting notified when
> you reply.
>
>
> I think you're doing something very specific since you're building on
> target and tweaking the build system so to help you more I would need to
> take a look at the recipes you are using.
>
>
> Alejandro
>
> On 05/04/2018 01:19 AM, Francesco Giancane wrote:
>
> Hi Alejandro,
>
> Thank you for your response.
> I know that openblas recipe is a pain, given that its Makefile is not
> correctly written with respect to cross-compilation. However, the library
> has been built locally on the target (sigh!) and then it is being deployed
> with a recipe which unpacks the built binary and copy them into the sys
> root.
>
> So, from the openblas side, everything is OK.
>
> What is actually wrong is the configuration step for python3-numpy and
> python3-scipy: in the setup.py they are instructed to search for some
> libraries (namely, openblas amongst them) but they fail to search for them
> given that the path they are searching for is the sysroot-native, rather
> than sysroot. (you can see it clearly from the logs attached in the first
> message).
>
> Installing the packages using pip3 install scipy numpy on the target
> results on a correct execution (and libraries are properly found and
> linked).
>
> Thank you,
> Francesco Giancane
>
>
>
>
> Il giorno ven 4 mag 2018 alle ore 03:01 Alejandro Enedino Hernandez
> Samaniego  ha scritto:
>
>> Hello Francesco,
>>
>>
>>
>> It would be helpful if you could provide more information about this,
>> e.g. access to your layer or your recipes, to make sure they are actually
>> being built correctly, also I would start by manually linking it against
>> the built library to see if the error is just not having them on the
>> correct path, but I seem to remember a lot of issues when trying to cross
>> compile openblas, specially for an arm MACHINE.
>>
>>
>>
>> Alejandro
>>
>>
>>
>> *From:* openembedded-core-boun...@lists.openembedded.org [mailto:
>> openembedded-core-boun...@lists.openembedded.org] *On Behalf Of *Francesco
>> Giancane
>> *Sent:* Wednesday, May 2, 2018 2:33 PM
>> *To:* OE-core 
>> *Subject:* [OE-core] python recipes with extensions fail to find/link
>> with proper libraries
>>
>>
>>
>> Hi Everyone,
>>
>> I found a problem working with python(3) and C extensions.
>>
>>
>>
>> The use-case is the recipe for python(3)-numpy, which can be linked
>> against a BLAS/LAPACK library to use accelerated matricial and vectorial
>> operations.
>>
>>
>>
>> As it is currently shipped, the recipe fails to find any proper library
>> and then does not use extensions.
>>
>>
>>
>> I wrote a custom recipe for openblas which is correctly built & deploy
>> using standard commands.
>>
>> openblas 

Re: [OE-core] python recipes with extensions fail to find/link with proper libraries

[Alejandro Enedino Hernandez Samaniego]

Hello Francesco,


Please hit the reply-all button, otherwise I wont be getting notified 
when you reply.



I think you're doing something very specific since you're building on 
target and tweaking the build system so to help you more I would need to 
take a look at the recipes you are using.



Alejandro


On 05/04/2018 01:19 AM, Francesco Giancane wrote:

Hi Alejandro,

Thank you for your response.
I know that openblas recipe is a pain, given that its Makefile is not 
correctly written with respect to cross-compilation. However, the 
library has been built locally on the target (sigh!) and then it is 
being deployed with a recipe which unpacks the built binary and copy 
them into the sys root.


So, from the openblas side, everything is OK.

What is actually wrong is the configuration step for python3-numpy and 
python3-scipy: in the setup.py they are instructed to search for some 
libraries (namely, openblas amongst them) but they fail to search for 
them given that the path they are searching for is the sysroot-native, 
rather than sysroot. (you can see it clearly from the logs attached in 
the first message).


Installing the packages using pip3 install scipy numpy on the target 
results on a correct execution (and libraries are properly found and 
linked).


Thank you,
Francesco Giancane




Il giorno ven 4 mag 2018 alle ore 03:01 Alejandro Enedino Hernandez 
Samaniego > ha scritto:


Hello Francesco,

It would be helpful if you could provide more information about
this, e.g. access to your layer or your recipes, to make sure they
are actually being built correctly, also I would start by manually
linking it against the built library to see if the error is just
not having them on the correct path, but I seem to remember a lot
of issues when trying to cross compile openblas, specially for an
arm MACHINE.

Alejandro

*From:* openembedded-core-boun...@lists.openembedded.org

[mailto:openembedded-core-boun...@lists.openembedded.org
] *On
Behalf Of *Francesco Giancane
*Sent:* Wednesday, May 2, 2018 2:33 PM
*To:* OE-core >
*Subject:* [OE-core] python recipes with extensions fail to
find/link with proper libraries

Hi Everyone,

I found a problem working with python(3) and C extensions.

The use-case is the recipe for python(3)-numpy, which can be
linked against a BLAS/LAPACK library to use accelerated matricial
and vectorial operations.

As it is currently shipped, the recipe fails to find any proper
library and then does not use extensions.

I wrote a custom recipe for openblas which is correctly built &
deploy using standard commands.

openblas is also added as a DEPENDS in a bbappend file for numpy.

However, the do_compile phase cannot find the aforementioned library.

Log attached:

DEBUG: Executing shell function do_compile

blas_opt_info:

blas_mkl_info:

customize UnixCCompiler

  libraries mkl_rt not found in

['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']

  NOT AVAILABLE

blis_info:

customize UnixCCompiler

  libraries blis not found in

['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']

  NOT AVAILABLE

openblas_info:

customize UnixCCompiler

customize UnixCCompiler

  libraries openblas not found in

['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']

  NOT AVAILABLE

atlas_3_10_blas_threads_info:

Setting PTATLAS=ATLAS

customize UnixCCompiler

  libraries tatlas not found in

['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']

  NOT AVAILABLE

atlas_3_10_blas_info:

customize UnixCCompiler

  libraries satlas not found in

['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']

  NOT AVAILABLE

atlas_blas_threads_info:

Setting PTATLAS=ATLAS

customize UnixCCompiler

  libraries ptf77blas,ptcblas,atlas not found in

['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']

  NOT AVAILABLE

atlas_blas_info:

customize UnixCCompiler

  libraries f77blas,cblas,atlas not found in

Re: [OE-core] python recipes with extensions fail to find/link with proper libraries

[Francesco Giancane]

Hi Alejandro,

Thank you for your response.
I know that openblas recipe is a pain, given that its Makefile is not
correctly written with respect to cross-compilation. However, the library
has been built locally on the target (sigh!) and then it is being deployed
with a recipe which unpacks the built binary and copy them into the sys
root.

So, from the openblas side, everything is OK.

What is actually wrong is the configuration step for python3-numpy and
python3-scipy: in the setup.py they are instructed to search for some
libraries (namely, openblas amongst them) but they fail to search for them
given that the path they are searching for is the sysroot-native, rather
than sysroot. (you can see it clearly from the logs attached in the first
message).

Installing the packages using pip3 install scipy numpy on the target
results on a correct execution (and libraries are properly found and
linked).

Thank you,
Francesco Giancane




Il giorno ven 4 mag 2018 alle ore 03:01 Alejandro Enedino Hernandez
Samaniego  ha scritto:

> Hello Francesco,
>
>
>
> It would be helpful if you could provide more information about this, e.g.
> access to your layer or your recipes, to make sure they are actually being
> built correctly, also I would start by manually linking it against the
> built library to see if the error is just not having them on the correct
> path, but I seem to remember a lot of issues when trying to cross compile
> openblas, specially for an arm MACHINE.
>
>
>
> Alejandro
>
>
>
> *From:* openembedded-core-boun...@lists.openembedded.org [mailto:
> openembedded-core-boun...@lists.openembedded.org] *On Behalf Of *Francesco
> Giancane
> *Sent:* Wednesday, May 2, 2018 2:33 PM
> *To:* OE-core 
> *Subject:* [OE-core] python recipes with extensions fail to find/link
> with proper libraries
>
>
>
> Hi Everyone,
>
> I found a problem working with python(3) and C extensions.
>
>
>
> The use-case is the recipe for python(3)-numpy, which can be linked
> against a BLAS/LAPACK library to use accelerated matricial and vectorial
> operations.
>
>
>
> As it is currently shipped, the recipe fails to find any proper library
> and then does not use extensions.
>
>
>
> I wrote a custom recipe for openblas which is correctly built & deploy
> using standard commands.
>
> openblas is also added as a DEPENDS in a bbappend file for numpy.
>
>
>
> However, the do_compile phase cannot find the aforementioned library.
>
>
>
> Log attached:
>
>
>
> DEBUG: Executing shell function do_compile
>
> blas_opt_info:
>
> blas_mkl_info:
>
> customize UnixCCompiler
>
>   libraries mkl_rt not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> blis_info:
>
> customize UnixCCompiler
>
>   libraries blis not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> openblas_info:
>
> customize UnixCCompiler
>
> customize UnixCCompiler
>
>   libraries openblas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_3_10_blas_threads_info:
>
> Setting PTATLAS=ATLAS
>
> customize UnixCCompiler
>
>   libraries tatlas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_3_10_blas_info:
>
> customize UnixCCompiler
>
>   libraries satlas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_blas_threads_info:
>
> Setting PTATLAS=ATLAS
>
> customize UnixCCompiler
>
>   libraries ptf77blas,ptcblas,atlas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_blas_info:
>
> customize UnixCCompiler
>
>   libraries f77blas,cblas,atlas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> blas_info:
>
> customize UnixCCompiler
>
>   libraries blas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> blas_src_info:
>
>   NOT AVAILABLE
>
>
>
>   NOT AVAILABLE
>
>
>
> ...And it is correct for it to fail as it is searching for openblas in the
> wrong path (sysroot-native vs sysroot).
>
>
>
> I would assume that this is a setuptools.bbclass issue, does anyone know
> what to do? Is it trivial to fix?
>
>
>
> I noticed that all the shipped python 

Re: [OE-core] python recipes with extensions fail to find/link with proper libraries

[Alejandro Enedino Hernandez Samaniego]

Hello Francesco,

It would be helpful if you could provide more information about this, e.g. 
access to your layer or your recipes, to make sure they are actually being 
built correctly, also I would start by manually linking it against the built 
library to see if the error is just not having them on the correct path, but I 
seem to remember a lot of issues when trying to cross compile openblas, 
specially for an arm MACHINE.

Alejandro

From: openembedded-core-boun...@lists.openembedded.org 
[mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of 
Francesco Giancane
Sent: Wednesday, May 2, 2018 2:33 PM
To: OE-core 
Subject: [OE-core] python recipes with extensions fail to find/link with proper 
libraries

Hi Everyone,
I found a problem working with python(3) and C extensions.

The use-case is the recipe for python(3)-numpy, which can be linked against a 
BLAS/LAPACK library to use accelerated matricial and vectorial operations.

As it is currently shipped, the recipe fails to find any proper library and 
then does not use extensions.

I wrote a custom recipe for openblas which is correctly built & deploy using 
standard commands.
openblas is also added as a DEPENDS in a bbappend file for numpy.

However, the do_compile phase cannot find the aforementioned library.

Log attached:

DEBUG: Executing shell function do_compile
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
  libraries mkl_rt not found in 
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

blis_info:
customize UnixCCompiler
  libraries blis not found in 
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

openblas_info:
customize UnixCCompiler
customize UnixCCompiler
  libraries openblas not found in 
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries tatlas not found in 
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_info:
customize UnixCCompiler
  libraries satlas not found in 
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in 
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

atlas_blas_info:
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in 
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

blas_info:
customize UnixCCompiler
  libraries blas not found in 
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

blas_src_info:
  NOT AVAILABLE

  NOT AVAILABLE

...And it is correct for it to fail as it is searching for openblas in the 
wrong path (sysroot-native vs sysroot).

I would assume that this is a setuptools.bbclass issue, does anyone know what 
to do? Is it trivial to fix?

I noticed that all the shipped python extensions were built as if we were 
building for x86, even though the MACHINE is an arm target.

Thank you for your attention,

Francesco Giancane

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] python recipes with extensions fail to find/link with proper libraries

[Francesco Giancane]

Hi Everyone,
I found a problem working with python(3) and C extensions.

The use-case is the recipe for python(3)-numpy, which can be linked against
a BLAS/LAPACK library to use accelerated matricial and vectorial operations.

As it is currently shipped, the recipe fails to find any proper library and
then does not use extensions.

I wrote a custom recipe for openblas which is correctly built & deploy
using standard commands.
openblas is also added as a DEPENDS in a bbappend file for numpy.

However, the do_compile phase cannot find the aforementioned library.

Log attached:

DEBUG: Executing shell function do_compile
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
  libraries mkl_rt not found in
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

blis_info:
customize UnixCCompiler
  libraries blis not found in
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

openblas_info:
customize UnixCCompiler
customize UnixCCompiler
  libraries openblas not found in
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries tatlas not found in
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_info:
customize UnixCCompiler
  libraries satlas not found in
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

atlas_blas_info:
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

blas_info:
customize UnixCCompiler
  libraries blas not found in
['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
  NOT AVAILABLE

blas_src_info:
  NOT AVAILABLE

  NOT AVAILABLE

...And it is correct for it to fail as it is searching for openblas in the
wrong path (sysroot-native vs sysroot).

I would assume that this is a setuptools.bbclass issue, does anyone know
what to do? Is it trivial to fix?

I noticed that all the shipped python extensions were built as if we were
building for x86, even though the MACHINE is an arm target.

Thank you for your attention,

Francesco Giancane
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core