On Mon, Nov 27, 2023 at 2:10 PM Dr. Thomas Orgis < thomas.or...@uni-hamburg.de> wrote:
> Hi, > > I'm involved in packaging NumPy for http://pkgsrc.org/. We install a > set of possible BLAS/CBLAS/LAPACK/LAPACKE packages side-by-side in the > same prefix. This includes multiple variants of OpenBLAS with regard to > multithreading (and indexing). For this purpose, we point software to > use the build-time chosen BLAS implementation via BLAS_LIBS and similar > variables, or, as seems to be appropriate for the new meson build of > NumPy, via .pc files. > > A package depends on the generic BLAS library family and central user > configuration chooses which one the packages should use during build. > > What would be the correct way to force the NumPy build to just use our > BLAS choice, avoiding any automatisms that might surprise us? > > How agnostic is NumPy regarding the offered BLAS? Does it have to know > that it is using OpenMP-parallelized OpenBLAS vs. the serial one (I'd > imagine just setting OMP_NUM_THREADS handles paralellism) The NumPy build does not know anything about this. It will just build, and it will simply call the OpenBLAS functionality - whether those execute under the hood in parallel or not, or with OpenMP or pthreads, is unknown. When a user or downstream library wants to control that parallelism, they can use an environment variable or https://github.com/joblib/threadpoolctl. or MKL, Netlib, … ? By default, the NumPy build will try all of those, in the order given by the `blas-order` and `lapack-order` build options (see `meson_options.txt in the root of the repo). It doesn't scale to have to tell it "openblas" or "netlib", > as there is no universal vocabulary to name the variants, and NumPy > doesn't even know openblas_openmp from serial openblas or > openblas_pthread (right?). > > Basically, I want to do > > meson setup -Dcblas_pc=$CBLAS_PC > > with CBLAS_PC being the module name of one of > > $prefix/lib/pkgconfig/cblas.pc > $prefix/lib/pkgconfig/openblas_pthread.pc > $prefix/lib/pkgconfig/openblas_openmp.pc > $prefix/lib/pkgconfig/openblas64_openmp.pc > > so that pkg-config does its thing without the NumPy build guessing > around. Is that feasible already? Is it easily supportable with some > changes to the build? I dream of a world where package build scripts > don't have to add dozens of idiosyncratic lines to detect these libs. > Yes, that is possible. You should be building with a build frontend (pip or pypa/build) and then the invocation will include `-C-Dblas=<your .pc name> -C-Dlapack=<your .pc name>`. See http://scipy.github.io/devdocs/building/blas_lapack.html for more guidance. > I'd like things to work like for CMake's FindBLAS with > -DBLA_PREFER_PKGCONFIG and -DBLA_PKGCONFIG_BLAS=$BLAS_PC (see > > https://cmake.org/cmake/help/latest/module/FindBLAS.html > > since version 3.25). > > Can we have that? > Yes, that is implemented since NumPy 1.26.2 and in the main branch. > > And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK? > No need for LAPACKE. > These are differing libraries, possibly coming in differing binaries, > even if your OpenBLAS builds also combine them. So I guess it should be > -Dcblas_pc and -Dlapacke_pc, both being possibly identical. A build of > the reference Netlib implementation provides four distinct libraries > and .pc files: > > $prefix/lib/pkgconfig/cblas.pc > $prefix/lib/pkgconfig/blas.pc > $prefix/lib/pkgconfig/lapacke.pc > $prefix/lib/pkgconfig/lapack.pc > > We do support installing openblas64 and friends alongside the others > and I imagine just setting an ILP64 option and repective symbol suffix > (none as of yet, as it's not a settled thing upstream) for the NumPy > build if a 64 variant is chosen by the user. I wonder a bit if there > are possible pitfalls combining other libraries with Python and > indirectly some incompatible BLAS variant via NumPy … but one point of > our user choice is that they could ensure that all packages really use > the same BLAS. > You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to craft a blas.pc which points at a 64-bit BLAS. Cheers, Ralf > > > Alrighty then, > > Thomas > > -- > Dr. Thomas Orgis > HPC @ Universität Hamburg > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: ralf.gomm...@googlemail.com >
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
