OpenBLAS and INTERFACE64=1
Dear List,
During the course of creating a port for OpenMolcas, I discovered that
it requires the 64-bit interface to OpenBLAS. My understanding is that
this promotes INTEGER to be 64-bit, so that arrays longer than 2**32-1
are supported. It was easy enough to shoe-horn in INTERFACE64=1 to
OpenBLAS:
maths.scm (openblas):
-
,@(let ((system (or (%current-target-system) (%current-system
(cond
((string-prefix? "i686" system)
'("DYNAMIC_ARCH=1"))
((string-prefix? "x86_64" system)
'("DYNAMIC_ARCH=1" "INTERFACE64=1"))
;; On MIPS we force the "SICORTEX" TARGET, as for the other
;; two available MIPS targets special extended instructions
;; for Loongson cores are used.
((string-prefix? "mips" system)
'("TARGET=SICORTEX" "INTERFACE64=1"))
;; On aarch64 force the generic 'armv8-a' target
((string-prefix? "aarch64" system)
'("TARGET=ARMV8" "INTERFACE64=1"))
(else '()
and OpenMolcas compiles and passes all tests.
I then set about to ensure that dependents also compiled without issue,
and in fact all do (at least, no *new* failures) except for python-numpy
and python-scipy, which Segmentation Fault on at least one test when
linked with an OpenBLAS defined with 64-bit integers.
Of course, it would be nice if each package can build, and also be
possible for Fortran routines to handle large arrays.
A few thoughts come to mind, perhaps in combination:
* add a variant to the current openblas package
* add compiler flags such as -fdefault-integer-8 to gfortran
* create a new package, e.g. openblas64, perhaps with a suffix to the
library name which distinguishes a 64-bit version from a 32-bit
version
(I'd like to have e.g. python-scipy and OpenMolcas on my machine at the
same time, so I'm worried about collisions in case 32-bit must be
present sometimes. Though on my 64-bit bit machines, I can say that I
never want 32-bit limitations of data size.)
On the web, people have mentioned all of the above, even adding a suffix
"_64" to each function in the API. I just don't see how this would be
workable given the number of packages that Guix aims to deliver.
Just a few thoughts which rekado has asked me to post to this list.
Hopefully someone has some ideas about the optimal way to proceed.
Best regards,
Eric
Re: Joint statement on the GNU Project
Ludovic Courtès writes: > Hello Guix! > > We, a group of GNU maintainers sharing a vision for a stronger GNU > Project, are publishing this statement today: > > https://guix.gnu.org/blog/2019/joint-statement-on-the-gnu-project/ > > We are somewhat abusing the Guix blog here, for lack of a better place, > but OTOH the future of GNU is obviously relevant to Guix. (Ricardo and > I started this initiative before Tobias, Maxim, and Marius were > on-board.) > > This mailing list is maybe not the best place to discuss this in detail > but your local GNU maintainers will surely be happy to answer any > questions you may have. :-) > > Ludo’. Don't know if I can add much to the volume of sentiment already expressed, but count me among those whose financial and moral support you have lost. I don't know who the next "Chief GNUsance" will be, but I know how s/he will be taken out.
Re: LAPACK vs. OpenBLAS
Ludovic Courtès writes: > Hello! > There are two cases: > > 1. Packages that depend on both ‘lapack’ and ‘openblas’. This is > almost always a mistake because ‘openblas’ provides the same > interfaces as ‘lapack’, only with a more efficient implementation. > > 2. Packages that depend on ‘lapack’ only: for the same reason, they > should depend on ‘openblas’ instead. My understanding is that Julia 1.6 (already released) requires 64-bit BLAS, and openblas-ilp64 should do the trick. There are scenarios where reference BLAS is more stable and/or reproducible than optimized BLAS libraries. Sometimes its lack of knowledge of OpenBLAS and/or a numerical issue. (cf. GNU R's discussion of which BLAS to use.)
Re: LAPACK vs. OpenBLAS
Hi Ludo'! Ludovic Courtès writes: > Hi Eric, > > Eric Brown skribis: > > Are there other cases where netlib BLAS is considered more appropriate > than OpenBLAS because it’s more numerically stable? > Sorry about that, the full discussion is here: https://cran.r-project.org/doc/manuals/r-devel/R-admin.html#Linear-algebra To answer your question on other examples: I would have to do some digging for the exact details, but there was once a time when LAPACK 3.x had an SVD that would solve with QR, most expensive but would give an answer for the "most singular" of problems. It is not, however, the "published" LAPACK algorithm and is not available. (A classic case where this occurs is when throwing junk into a linear regression, whose solution is bullet-proof with the SVD. Very important for R, and many more langs and other problems such as factor analysis.) A few random points: * Good/improving compilers gfortran,gcc often make the reference lapack/blas fast enough. IIRC it is good as any optimized BLAS for small matrices. If reporting e.g. convergence issues, people converge here for apples-to-apples. * ATLAS can empirically tune for architectures that are not getting love by the OpenBLAS team. So, non-amd64 has *something* more performant than Reference LAPACK/BLAS. I've often seen distributed binaries have to choose something lackluster to satisfy older processors. * I usually use OpenBLAS because it also gives SMP. I notice it doesn't cover all the archs that Guix covers, but I think it should be default go-to unless of problems for x86_64 at least. * If memory serves, layering multiple LAPACKs came from the days when the optimized BLAS's where incomplete. People just slapped -llapack on the end to make sure that if the optimized lib didn't get it then lapack would, just that would be only at Fortran speed :-( lol Best regards, Eric
Re: LAPACK vs. OpenBLAS
Ludovic Courtès writes: > Hi Eric, > > Eric Brown skribis: > >> There are scenarios where reference BLAS is more stable and/or >> reproducible than optimized BLAS libraries. Sometimes its lack of >> knowledge of OpenBLAS and/or a numerical issue. >> >> (cf. GNU R's discussion of which BLAS to use.) > > Do you have a pointer to that discussion? > > Are there other cases where netlib BLAS is considered more appropriate > than OpenBLAS because it’s more numerically stable? > > Thanks for your feedback, > Ludo’. Also: * More controversially: consider 64-bit addressing. It is depressing to hit 32-bit memory limits on large machines which ought to accomodate larger problems. (e.g. -fdefault-integer-8, ILP64)
Re: ️ Install every Guix package ️
kiasoc5 writes: > I just realized that this downloads all packages into the store, which not the > same as installing all the packages into one or many profiles. I wonder what > the > minimum set of profiles to install all packages while avoiding collisions > might > be? Correct, my reply was about building everything rather than installing everything. But--as a case merely to prove a point--if you include the kernel in the required set of packages, then the single set of non-colliding packages can not be defined because there are several mutually-exclusive kernels. A valid "maximum coverage" set must have some additional constraints.
Re: adding motif to guix
Andy Tai writes: > It should be; openmotif was just some kind of marketing term > > Although guix has some existing packages using lesstif, the now dead > early motif clone (as free software) and I don't know if emacs may be > using that... > Thanks for the clarification. My interest in this package is because openmotif is the best of the X window managers that do not suffer from a rather infamous and longstanding GTK bug which will bring down Emacs.
Re: ️ Install every Guix package ️
"jgart" writes: > Hi Guixers, > > How would you approach writing a script that installs every Guix package > exhaustively for your current revision? > > I'm thinking of something similar to `all-packages` on PyPi but for every > Guix package (the whole wide ️). > > https://pypi.org/project/all-packages/ guix package -A | cut -f 1 | sort -u | xargs guix build --keep-going can accomplish this. The `--keep-going' flag is useful in case a package does not build. Occasionally, I will sandwich in a negated grep to exclude packages: guix package -A | cut -f 1 | sort -u | grep -v "emacs-guile" | xargs guix build --keep-going (Aside: emacs-guile-on-guix-system is the pinnacle of GNU, IMO but it failed to build--appearing to be abandoned--and was removed.) One may also select just the packages they want, perhaps by using a scripting language such as python to make larger selections more convenient. This can be combined with a substitute server and an offload build machine to make an on-site cache. One can choose whether to employ the Guix substitute servers if that would be beneficial.
Re: ️ Install every Guix package ️
kiasoc5 writes: > Surely if there are outstanding questions that could be answered by installing > all packages, they could be answered in another way? I do this to keep packages as reasonably close to "locked and loaded" as possible (including all external channels) so that when I need them interactively, they are ready on my system. It's easier and less trouble than Cuirass, in my view.
Re: adding motif to guix
Andy Tai writes: > Hi, motif was a major component of Unix for many years and it would be > good to add the (now free software) motif to GNU Guix. Can package > reviewers take a look of the patch in this issue > > https://issues.guix.gnu.org/59092 > > Thanks Hello Andy: By chance is this the same as `openmotif' that is found in the likes of Debian, also referenced by programs such as GNU Emacs (--with-x-toolkit) ? Thanks, Eric
