(I took this off-list unintentionally, so I'm forward each email to the
list now)
---------- Forwarded message ---------
From: Ralf Gommers <ralf.gomm...@gmail.com>
Date: Thu, Dec 28, 2023 at 8:51 PM
Subject: Re: incomplete BLAS/CBLAS linking (Telling meson build which
CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module)
To: Dr. Thomas Orgis <thomas.or...@uni-hamburg.de>
On Mon, Dec 25, 2023 at 10:37 AM Dr. Thomas Orgis <
thomas.or...@uni-hamburg.de> wrote:
> Hapy holidays … but I have an issue still that hopefully can be
> addressed with the meson blas detection you are upstreaming(?).
>
Happy holidays to you too. And yes, I hope so:)
>
> Am Wed, 6 Dec 2023 18:06:01 +0100
> schrieb Ralf Gommers <ralf.gomm...@gmail.com>:
>
> > > Well, now is another day. Pkgsrc uses python -m build and I added
> > >
> > > -Csetup-args=-Dblas=${CBLAS_PC}
> -Csetup-args=-Dlapack=${LAPACK_PC}
> > >
> > > which seems to work out fine using cblas.pc and lapack.pc in the case
> > > of the netlib install. In fact, most linking is done only to libblas.so
> > > instead of libcblas.so, as the linker is smart enough to throw away the
> > > unused lib.
> > >
> >
> > Great, thanks for confirming!
>
> This works for numpy and also installs scipy nicely, but this produces
> a broken scipy install when using netlib reference libraries from
> pkgsrc. These come as
>
> libblas.so
> liblapack.so (NEEDing libblas.so)
> libcblas.so (NEEDing libblas.so)
> libpapacke.so (NEEDing liblapack.so, hence libblas.so)
>
> and their respective .pc files. This is the natural order that occus to
> me when building from netlib upstream.
This should work fine. It's auto-detected in NumPy already, and will be in
SciPy in the future. For now, using `-Dblas=blas -Dlapack=lapack` in the
SciPy build should work.
> This also means that one could
> just replace BLAS and put stock LAPACK on top, what optimized BLAS libs
> usually start out with.
This is indeed possible, if unusual. It's supported and one reason for why
we have separate `blas` and `lapack` flags. I'd discourage distros from
shipping something like that by default though, since it tends to lead to
problems. Arch Linux used to do this, shipping an OpenBLAS without LAPACK
symbols. Luckily they finally fixed that. Shipping non-default build
configs like that is invariably a bad idea, and should only be done if
there's a pressing need.
Only that they tend to pack all symbols into
> one common library, which then project builds like numpy rely on.
>
> Telling the meson build that BLAS is libcblas works as long as actually
> CBLAS symbols are used.
Please never do this. The library is BLAS, so you should use `-Dblas=blas`
for NumPy. It will find `cblas` just fine that way.
> If not — I presume now, as I didn't yet see the
> actual build lines that are triggered via the python -m build and meson
> indirections — the linker might discard the -lcblas and leave symbols
> unresolved (--as-needed but no --no-undefined).
>
> This happens with scipy:
>
> $ LANG=C readelf -d
> /data/pkg/lib/python3.11/site-packages/scipy/sparse/linalg/_dsolve/_superlu.so
> |grep NEEDED
> 0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>
This is probably a bug in SciPy. The build target depends on both `blas`
and `lapack`, and sets `DUSE_VENDOR_BLAS=1`. However, it looks like it
should depend on `cblas`. If you add `cblas` to this line, I think it'll
fix the issue:
https://github.com/scipy/scipy/blob/6452a48c9611d16140b160091de6cf5299fadd9f/scipy/sparse/linalg/_dsolve/meson.build#L208
.
> It would link against libopenblas_openmp.so if that had been the CBLAS
> (and LAPACK) choice and all would be fine, but here, it should link
> with libcblas.so, or directly to libblas.so, just like our regular
> install of superlu:
>
> $ LANG=C readelf -d /data/pkg/lib/libsuperlu.so|grep NEEDED
> 0x0000000000000001 (NEEDED) Shared library: [libblas.so.3]
> 0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>
> Of course, just not vendoring superlu would be one solution for scipy,
> but I think the deeper issue with the meson BLAS support should be
> solved: The 4 parts of the BLAS canon (not talking about SCALAPACK etc.
> yet) need to be handled explicitly.
>
> It is confusing, though, as meson prints this:
>
> Run-time dependency blas found: YES 3.11.0
> Run-time dependency cblas found: YES 3.11.0
> Run-time dependency lapack found: YES 3.11.0
>
> It suggests that it looked for and found 3 libraries, but actually, it
> only cared for -llapack and -lcblas. It needs to find -lblas directly,
> too (or the cblas package separately, for that matter, not as component
> of blas).
>
> Is that easily fixable from your side? (I'm assuming numpy, scipy and
> the future stock BLAS support of meson are handled together.) Is this
> just an oversight on the scipy side and they could link the vendored
> superly with -lblas without changing the meson machinery?
>
Yes, I think it's an oversight indeed, and we didn't notice because we have
very limited CI for BLAS/LAPACK configs still for SciPy (just a couple of
ATLAS usages next to the OpenBLAS default, no jobs with
Netlib/MKL/Accelerate).
Cheers,
Ralf
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
