Re: Allowing multiple providers of libraries - BLAS/LAPACK
Am 29.10.20 um 22:51 schrieb Aisha Tammy: > On 10/29/20 3:47 PM, Martin Reindl wrote: >> Am 29.10.20 um 00:07 schrieb Aisha Tammy: >>> Attached updated diff with documentation generated using doxygen. >> >> Can you please provide a diff which applies with 'patch -p1' (or -p0)? >> Thank you. >> >> -m >> > I've reattached a patch. > It seems thunderbird added some more whitespaces. Thanks, Some quick comments: - extra @man man/man3/_usr_obj_ports_lapack-3.9.0_lapack-3.9.0* in PLIST - conflict markers are needed in PLIST, there should be an upgrade of some kind from cblas/blas - for easier testing and review I would suggest an incremental approach: 1. update to 3.9.0 2. switch to doxygen generated manpages 3. replace blas/cblas -m
Re: Allowing multiple providers of libraries - BLAS/LAPACK
Am 29.10.20 um 00:07 schrieb Aisha Tammy: > Attached updated diff with documentation generated using doxygen. Can you please provide a diff which applies with 'patch -p1' (or -p0)? Thank you. -m
Re: Allowing multiple providers of libraries - BLAS/LAPACK
On 10/28/20 5:46 PM, Aisha Tammy wrote: On 10/28/20 3:31 PM, Ingo Schwarze wrote: Hi Aisha, Aisha Tammy wrote on Wed, Oct 28, 2020 at 01:12:15PM -0400: Here is a preliminary version update of math/lapack to 3.9.0 with some patches from debian and gentoo. While FORTAN used to be my favourite language until about 1998, i got out of the habit of using it after that, so i can't really comment on the port in general, but ... Current caveats (1) I've had to remove the man pages installation. 1. Why? In general, if a major change is needed when updating a port, it is useful to mention the reason. 2. Isn't that a blocker? For a programming library, having section 3 documentation seems crucial to me. What is the point of an undocumented public API? Yes, i am aware that lapack is among the last 20 or so ports that require USE_GROFF, and among the worst handful of these because the man(7) code is both unusually problematic from a qualitative standpoint (.ti nested inside .TP head, ...) and also hard to handle from a perspective of quantity - the library is so large that even reviewing the manual pages regarding which problems are contained in them isn't easy. But none of that should be your problem, nor should it prevent formatting the manual pages with groff and installing them. Yours, Ingo That's fair. I removed them as it seemed like none of the other distributions are installing them in this package. Some of them have a separate man-pages packages for lapack. Other reason was that the man pages haven't been updated from 2007. So I am also not sure if they are the most up to date. But as you said, having them is better than not having them. I can try to add them back but it might take some more time. Aisha Just as I sent this email, I noticed that it seems they have a Doxygen man generator that is quite up to date. - https://github.com/Reference-LAPACK/lapack/blob/master/DOCS/Doxyfile_man Their prebuilt man files are from 2007 so I'll try to use this to generate the man pages! Cheers, Aisha
Re: Allowing multiple providers of libraries - BLAS/LAPACK
On 10/28/20 3:31 PM, Ingo Schwarze wrote: Hi Aisha, Aisha Tammy wrote on Wed, Oct 28, 2020 at 01:12:15PM -0400: Here is a preliminary version update of math/lapack to 3.9.0 with some patches from debian and gentoo. While FORTAN used to be my favourite language until about 1998, i got out of the habit of using it after that, so i can't really comment on the port in general, but ... Current caveats (1) I've had to remove the man pages installation. 1. Why? In general, if a major change is needed when updating a port, it is useful to mention the reason. 2. Isn't that a blocker? For a programming library, having section 3 documentation seems crucial to me. What is the point of an undocumented public API? Yes, i am aware that lapack is among the last 20 or so ports that require USE_GROFF, and among the worst handful of these because the man(7) code is both unusually problematic from a qualitative standpoint (.ti nested inside .TP head, ...) and also hard to handle from a perspective of quantity - the library is so large that even reviewing the manual pages regarding which problems are contained in them isn't easy. But none of that should be your problem, nor should it prevent formatting the manual pages with groff and installing them. Yours, Ingo That's fair. I removed them as it seemed like none of the other distributions are installing them in this package. Some of them have a separate man-pages packages for lapack. Other reason was that the man pages haven't been updated from 2007. So I am also not sure if they are the most up to date. But as you said, having them is better than not having them. I can try to add them back but it might take some more time. Aisha
Re: Allowing multiple providers of libraries - BLAS/LAPACK
On 10/28/20 1:12 PM, Aisha Tammy wrote:
On 10/28/20 8:23 AM, Aisha Tammy wrote:
On 10/28/20 7:46 AM, Stuart Henderson wrote:
+cc some people who might be interested to make sure they see this :)
On 2020/10/28 07:32, Aisha Tammy wrote:
Hi,
I was wondering if it is possible to have multiple providers for a library
and then selecting one of them to be used, selecting which provider is
to be installed.
Case in point, we have the math/{lapack,blas} libraries, which are the
reference implementations and not optimized.
I am working on getting OpenBLAS built and ported, which can also
provide the libblas.so and liblapack.so (among other libraries).
It is ABI compatible with the latest BLAS/LAPACK standards so it should
ideally be a drop in replacement for the libraries for math/{lapack,blas}.
This way there is a significant runtime benefit for multiple science
libraries like numpy, scipy, eigen and a bunch of others.
This replacement mechanism is provided in Gentoo and Debian (and its
derivatives).
I don't think we can have the full replacement mechanism at runtime
due to the linking mechanism being different but it should at least
be possible during install time and relinking libraries.
I am not sure how to go about doing this in OpenBSD.
Is there any interest in doing this? I am hoping I can put something
together if people are interested in wanting OpenBLAS based libraries
Cheers,
Aisha
Having alternatives like this is similar to the case where a library has
flavours, which certainly adds complication - we mostly resist doing this
in ports, the exception is apr-util's ldap flavour and I wouldn't really
want to add more.
Is there likely to be a downside to switching outright to OpenBLAS?
Theoretically no, as it is supposed to be ABI compatible.
Reality is often disappointing.
The reference implementations focus on correctness while
OpenBLAS implementation focuses on speed.
So it is *possible* that there may be bugs in the OpenBLAS
based libraries (not saying that reference impl is perfect).
I agree that this will complicate things.
There are like 4 different providers of BLAS/LAPACK that I am
maintaining in Gentoo. I do not want to add all of them in
OpenBSD, OpenBLAS is by far the superior alternative to the
other 3.
The general method for development: write code using BLAS,
test with reference impl, then use in production with OpenBLAS.
One way in which I think this can be simplified:
Combine math/BLAS and math/LAPACK into one package - math/LAPACK
which can supply both libraries. Almost every package which needs
BLAS also needs LAPACK, so there is little configurability that
we lose here.
Then there can be two flavors math/LAPACK-reference and
math/LAPACK-openblas.
More fixes I want to add - add LAPACKE library as well.
For some reason, we don't seem to be building that one.
Best,
Aisha
Hi,
Here is a preliminary version update of math/lapack to 3.9.0
with some patches from debian and gentoo.
Current caveats
(1) I've had to remove the man pages installation.
(2) I am not sure how to enable tests without explicitly putting
the BUILD_TESTING=ON by default, but it adds ~3000 targets
on top of the default ~6700 targets, which is A LOT. But building
the port with BUILD_TESTING=ON and then doing make test works
and all 121 tests pass.
This would be the lapack-reference flavour later, if we do implement the
lapack/openblas thing.
In any case, this should be useful by itself.
Also, this obsoletes the math/blas and math/cblas packages.
This also adds building the lapacke library, which was not being done
before this.
Cheers
Aisha
git is based and dank af, cuz it does not add new files in git diff...
reattaching diff with new patches as well
Aisha
diff --git a/math/lapack/Makefile b/math/lapack/Makefile
index 85e2b1a3565..ef961bfe003 100644
--- a/math/lapack/Makefile
+++ b/math/lapack/Makefile
@@ -2,14 +2,17 @@
COMMENT= library of Fortran linear algebra subroutines
-VERSION= 3.8.0
-DISTNAME= lapack-${VERSION}
-REVISION= 1
+CATEGORIES=math devel
-SHARED_LIBS= lapack 7.1
+VERSION= 3.9.0
+GH_ACCOUNT=Reference-LAPACK
+GH_PROJECT=lapack
+GH_TAGNAME=v${VERSION}
-CATEGORIES= math
-DISTFILES= ${DISTNAME}.tar.gz manpages.tgz:0
+SHARED_LIBS= lapack 7.2 \
+ lapacke 7.2 \
+ blas 7.2 \
+ cblas 7.2
HOMEPAGE= http://www.netlib.org/lapack/
@@ -18,69 +21,15 @@ MAINTAINER= Steven Mestdagh
# BSD
PERMIT_PACKAGE=Yes
-MASTER_SITES= https://www.netlib.org/lapack/ \
- https://www.netlib.no/netlib/lapack/
-MASTER_SITES0= https://www.netlib.org/lapack/
-DIST_SUBDIR= ${DISTNAME}
+WANTLIB+= m
-LIB_DEPENDS = math/blas
-WANTLIB = blas>=1 m
-
-MODULES= fortran
+MODULES= devel/cmake fortran
MODFORTRAN_COMPILER = gfortran
BUILD_DEPENDS= ${MODFORTRAN_BUILD_DEPENDS}
-MAKE_ENV= SHLIB_MAJOR=${LIBlapack_VERSION:R} \
- SHLIB_MINOR=${LIBlapack_VERSION:E} \
-
Re: Allowing multiple providers of libraries - BLAS/LAPACK
On 10/28/20 8:23 AM, Aisha Tammy wrote:
On 10/28/20 7:46 AM, Stuart Henderson wrote:
+cc some people who might be interested to make sure they see this :)
On 2020/10/28 07:32, Aisha Tammy wrote:
Hi,
I was wondering if it is possible to have multiple providers for a library
and then selecting one of them to be used, selecting which provider is
to be installed.
Case in point, we have the math/{lapack,blas} libraries, which are the
reference implementations and not optimized.
I am working on getting OpenBLAS built and ported, which can also
provide the libblas.so and liblapack.so (among other libraries).
It is ABI compatible with the latest BLAS/LAPACK standards so it should
ideally be a drop in replacement for the libraries for math/{lapack,blas}.
This way there is a significant runtime benefit for multiple science
libraries like numpy, scipy, eigen and a bunch of others.
This replacement mechanism is provided in Gentoo and Debian (and its
derivatives).
I don't think we can have the full replacement mechanism at runtime
due to the linking mechanism being different but it should at least
be possible during install time and relinking libraries.
I am not sure how to go about doing this in OpenBSD.
Is there any interest in doing this? I am hoping I can put something
together if people are interested in wanting OpenBLAS based libraries
Cheers,
Aisha
Having alternatives like this is similar to the case where a library has
flavours, which certainly adds complication - we mostly resist doing this
in ports, the exception is apr-util's ldap flavour and I wouldn't really
want to add more.
Is there likely to be a downside to switching outright to OpenBLAS?
Theoretically no, as it is supposed to be ABI compatible.
Reality is often disappointing.
The reference implementations focus on correctness while
OpenBLAS implementation focuses on speed.
So it is *possible* that there may be bugs in the OpenBLAS
based libraries (not saying that reference impl is perfect).
I agree that this will complicate things.
There are like 4 different providers of BLAS/LAPACK that I am
maintaining in Gentoo. I do not want to add all of them in
OpenBSD, OpenBLAS is by far the superior alternative to the
other 3.
The general method for development: write code using BLAS,
test with reference impl, then use in production with OpenBLAS.
One way in which I think this can be simplified:
Combine math/BLAS and math/LAPACK into one package - math/LAPACK
which can supply both libraries. Almost every package which needs
BLAS also needs LAPACK, so there is little configurability that
we lose here.
Then there can be two flavors math/LAPACK-reference and
math/LAPACK-openblas.
More fixes I want to add - add LAPACKE library as well.
For some reason, we don't seem to be building that one.
Best,
Aisha
Hi,
Here is a preliminary version update of math/lapack to 3.9.0
with some patches from debian and gentoo.
Current caveats
(1) I've had to remove the man pages installation.
(2) I am not sure how to enable tests without explicitly putting
the BUILD_TESTING=ON by default, but it adds ~3000 targets
on top of the default ~6700 targets, which is A LOT. But building
the port with BUILD_TESTING=ON and then doing make test works
and all 121 tests pass.
This would be the lapack-reference flavour later, if we do implement the
lapack/openblas thing.
In any case, this should be useful by itself.
Also, this obsoletes the math/blas and math/cblas packages.
This also adds building the lapacke library, which was not being done
before this.
Cheers
Aisha
diff --git a/math/lapack/Makefile b/math/lapack/Makefile
index 85e2b1a3565..ef961bfe003 100644
--- a/math/lapack/Makefile
+++ b/math/lapack/Makefile
@@ -2,14 +2,17 @@
COMMENT= library of Fortran linear algebra subroutines
-VERSION= 3.8.0
-DISTNAME= lapack-${VERSION}
-REVISION= 1
+CATEGORIES=math devel
-SHARED_LIBS= lapack 7.1
+VERSION= 3.9.0
+GH_ACCOUNT=Reference-LAPACK
+GH_PROJECT=lapack
+GH_TAGNAME=v${VERSION}
-CATEGORIES= math
-DISTFILES= ${DISTNAME}.tar.gz manpages.tgz:0
+SHARED_LIBS= lapack 7.2 \
+ lapacke 7.2 \
+ blas 7.2 \
+ cblas 7.2
HOMEPAGE= http://www.netlib.org/lapack/
@@ -18,69 +21,15 @@ MAINTAINER= Steven Mestdagh
# BSD
PERMIT_PACKAGE=Yes
-MASTER_SITES= https://www.netlib.org/lapack/ \
- https://www.netlib.no/netlib/lapack/
-MASTER_SITES0= https://www.netlib.org/lapack/
-DIST_SUBDIR= ${DISTNAME}
+WANTLIB+= m
-LIB_DEPENDS = math/blas
-WANTLIB = blas>=1 m
-
-MODULES= fortran
+MODULES= devel/cmake fortran
MODFORTRAN_COMPILER = gfortran
BUILD_DEPENDS= ${MODFORTRAN_BUILD_DEPENDS}
-MAKE_ENV= SHLIB_MAJOR=${LIBlapack_VERSION:R} \
- SHLIB_MINOR=${LIBlapack_VERSION:E} \
- TIMER=EXT_ETIME \
- FC="${MODFORTRAN_COMPILER} -cpp" \
- CC=${MODFORTRAN_COMPILER}
-FAKE_FLAGS=LIBDIR=${LOCALBASE}/lib
Re: Allowing multiple providers of libraries - BLAS/LAPACK
Hi Aisha, Aisha Tammy wrote on Wed, Oct 28, 2020 at 01:12:15PM -0400: > Here is a preliminary version update of math/lapack to 3.9.0 > with some patches from debian and gentoo. While FORTAN used to be my favourite language until about 1998, i got out of the habit of using it after that, so i can't really comment on the port in general, but ... > Current caveats > (1) I've had to remove the man pages installation. 1. Why? In general, if a major change is needed when updating a port, it is useful to mention the reason. 2. Isn't that a blocker? For a programming library, having section 3 documentation seems crucial to me. What is the point of an undocumented public API? Yes, i am aware that lapack is among the last 20 or so ports that require USE_GROFF, and among the worst handful of these because the man(7) code is both unusually problematic from a qualitative standpoint (.ti nested inside .TP head, ...) and also hard to handle from a perspective of quantity - the library is so large that even reviewing the manual pages regarding which problems are contained in them isn't easy. But none of that should be your problem, nor should it prevent formatting the manual pages with groff and installing them. Yours, Ingo
Re: Allowing multiple providers of libraries - BLAS/LAPACK
On 10/28/20 7:46 AM, Stuart Henderson wrote:
+cc some people who might be interested to make sure they see this :)
On 2020/10/28 07:32, Aisha Tammy wrote:
Hi,
I was wondering if it is possible to have multiple providers for a library
and then selecting one of them to be used, selecting which provider is
to be installed.
Case in point, we have the math/{lapack,blas} libraries, which are the
reference implementations and not optimized.
I am working on getting OpenBLAS built and ported, which can also
provide the libblas.so and liblapack.so (among other libraries).
It is ABI compatible with the latest BLAS/LAPACK standards so it should
ideally be a drop in replacement for the libraries for math/{lapack,blas}.
This way there is a significant runtime benefit for multiple science
libraries like numpy, scipy, eigen and a bunch of others.
This replacement mechanism is provided in Gentoo and Debian (and its
derivatives).
I don't think we can have the full replacement mechanism at runtime
due to the linking mechanism being different but it should at least
be possible during install time and relinking libraries.
I am not sure how to go about doing this in OpenBSD.
Is there any interest in doing this? I am hoping I can put something
together if people are interested in wanting OpenBLAS based libraries
Cheers,
Aisha
Having alternatives like this is similar to the case where a library has
flavours, which certainly adds complication - we mostly resist doing this
in ports, the exception is apr-util's ldap flavour and I wouldn't really
want to add more.
Is there likely to be a downside to switching outright to OpenBLAS?
Theoretically no, as it is supposed to be ABI compatible.
Reality is often disappointing.
The reference implementations focus on correctness while
OpenBLAS implementation focuses on speed.
So it is *possible* that there may be bugs in the OpenBLAS
based libraries (not saying that reference impl is perfect).
I agree that this will complicate things.
There are like 4 different providers of BLAS/LAPACK that I am
maintaining in Gentoo. I do not want to add all of them in
OpenBSD, OpenBLAS is by far the superior alternative to the
other 3.
The general method for development: write code using BLAS,
test with reference impl, then use in production with OpenBLAS.
One way in which I think this can be simplified:
Combine math/BLAS and math/LAPACK into one package - math/LAPACK
which can supply both libraries. Almost every package which needs
BLAS also needs LAPACK, so there is little configurability that
we lose here.
Then there can be two flavors math/LAPACK-reference and
math/LAPACK-openblas.
More fixes I want to add - add LAPACKE library as well.
For some reason, we don't seem to be building that one.
Best,
Aisha
Re: Allowing multiple providers of libraries - BLAS/LAPACK
On 10/28/20 7:46 AM, Stuart Henderson wrote:
+cc some people who might be interested to make sure they see this :)
On 2020/10/28 07:32, Aisha Tammy wrote:
Hi,
I was wondering if it is possible to have multiple providers for a library
and then selecting one of them to be used, selecting which provider is
to be installed.
Case in point, we have the math/{lapack,blas} libraries, which are the
reference implementations and not optimized.
I am working on getting OpenBLAS built and ported, which can also
provide the libblas.so and liblapack.so (among other libraries).
It is ABI compatible with the latest BLAS/LAPACK standards so it should
ideally be a drop in replacement for the libraries for math/{lapack,blas}.
This way there is a significant runtime benefit for multiple science
libraries like numpy, scipy, eigen and a bunch of others.
This replacement mechanism is provided in Gentoo and Debian (and its
derivatives).
I don't think we can have the full replacement mechanism at runtime
due to the linking mechanism being different but it should at least
be possible during install time and relinking libraries.
I am not sure how to go about doing this in OpenBSD.
Is there any interest in doing this? I am hoping I can put something
together if people are interested in wanting OpenBLAS based libraries
Cheers,
Aisha
Having alternatives like this is similar to the case where a library has
flavours, which certainly adds complication - we mostly resist doing this
in ports, the exception is apr-util's ldap flavour and I wouldn't really
want to add more.
Is there likely to be a downside to switching outright to OpenBLAS?
Theoretically no, as it is supposed to be ABI compatible.
Reality is often disappointing.
The reference implementations focus on correctness while
OpenBLAS implementation focuses on speed.
So it is *possible* that there may be bugs in the OpenBLAS
based libraries (not saying that reference impl is perfect).
I agree that this will complicate things.
There are like 4 different providers of BLAS/LAPACK that I am
maintaining in Gentoo. I do not want to add all of them in
OpenBSD, OpenBLAS is by far the superior alternative to the
other 3.
The general method for development: write code using BLAS,
test with reference impl, then use in production with OpenBLAS.
One way in which I think this can be simplified:
Combine math/BLAS and math/LAPACK into one package - math/LAPACK
which can supply both libraries. Almost every package which needs
BLAS also needs LAPACK, so there is little configurability that
we lose here.
Then there can be two flavors math/LAPACK-reference and
math/LAPACK-openblas.
More fixes I want to add - add LAPACKE library as well.
For some reason, we don't seem to be building that one.
Best,
Aisha
Allowing multiple providers of libraries - BLAS/LAPACK
Hi,
I was wondering if it is possible to have multiple providers for a library
and then selecting one of them to be used, selecting which provider is
to be installed.
Case in point, we have the math/{lapack,blas} libraries, which are the
reference implementations and not optimized.
I am working on getting OpenBLAS built and ported, which can also
provide the libblas.so and liblapack.so (among other libraries).
It is ABI compatible with the latest BLAS/LAPACK standards so it should
ideally be a drop in replacement for the libraries for math/{lapack,blas}.
This way there is a significant runtime benefit for multiple science
libraries like numpy, scipy, eigen and a bunch of others.
This replacement mechanism is provided in Gentoo and Debian (and its
derivatives).
I don't think we can have the full replacement mechanism at runtime
due to the linking mechanism being different but it should at least
be possible during install time and relinking libraries.
I am not sure how to go about doing this in OpenBSD.
Is there any interest in doing this? I am hoping I can put something
together if people are interested in wanting OpenBLAS based libraries
Cheers,
Aisha
Re: Allowing multiple providers of libraries - BLAS/LAPACK
+cc some people who might be interested to make sure they see this :)
On 2020/10/28 07:32, Aisha Tammy wrote:
> Hi,
> I was wondering if it is possible to have multiple providers for a library
> and then selecting one of them to be used, selecting which provider is
> to be installed.
> Case in point, we have the math/{lapack,blas} libraries, which are the
> reference implementations and not optimized.
> I am working on getting OpenBLAS built and ported, which can also
> provide the libblas.so and liblapack.so (among other libraries).
> It is ABI compatible with the latest BLAS/LAPACK standards so it should
> ideally be a drop in replacement for the libraries for math/{lapack,blas}.
> This way there is a significant runtime benefit for multiple science
> libraries like numpy, scipy, eigen and a bunch of others.
> This replacement mechanism is provided in Gentoo and Debian (and its
> derivatives).
> I don't think we can have the full replacement mechanism at runtime
> due to the linking mechanism being different but it should at least
> be possible during install time and relinking libraries.
> I am not sure how to go about doing this in OpenBSD.
>
> Is there any interest in doing this? I am hoping I can put something
> together if people are interested in wanting OpenBLAS based libraries
>
> Cheers,
> Aisha
Having alternatives like this is similar to the case where a library has
flavours, which certainly adds complication - we mostly resist doing this
in ports, the exception is apr-util's ldap flavour and I wouldn't really
want to add more.
Is there likely to be a downside to switching outright to OpenBLAS?
