Hi Ole, You only need to rebuild "OpenBLAS/0.3.1-GCC-7.3.0-2.30". These are the commands:
$> wget https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/develop/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb $> wget https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/develop/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.3.1_disable-special-handling-of-OpenMP-thread-memory-allocation.patch $> eb ./OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb --force module "ScaLAPACK/2.0.2-gompi-2018b-OpenBLAS-0.3.1" only provides a static library "libscalapack.a" so you should also rebuild it after you have update your OpenBLAS installation. In principle every other module that does dynamic linking with OpenBLAS should be fixed and use the new library. The problem could be if any of your existing modules did static linking with openblas and the binary includes the old buggy openblas library. e.g. in my system I had to rebuild this one https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/j/JAGS/JAGS-4.3.0-foss-2018b.eb I hope it helps. regards, Pablo. On Wed, May 29, 2019 at 9:47 AM Ole Holm Nielsen <[email protected]> wrote: > > Hi Pablo, > > Thanks for the good news! For those of us who are not experts, could > you kindly describe the commands required to update our currently > installed OpenBLAS modules correctly? Are there any caveats with > updating an existing module in a toolchain (foss)? > > We currently have these OpenBLAS modules installed: > > $ ml av OpenBLAS/ > > -------------------------- /home/modules/modules/all > --------------------------- > OpenBLAS/0.2.20-GCC-6.4.0-2.28 OpenBLAS/0.3.5-GCC-8.2.0-2.31.1 (D) > OpenBLAS/0.3.1-GCC-7.3.0-2.30 > > So is the update command simply this one? > > eb --rebuild OpenBLAS-0.3.5-GCC-8.2.0-2.31.1.eb > OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb OpenBLAS-0.2.20-GCC-6.4.0-2.28.eb > > Is the --force flag also required? > > I suppose that we do not have to rebuild the ScaLAPACK modules as well? > > Thanks a lot, > Ole > > > > On 5/28/19 5:14 PM, Pablo Escobar Lopez wrote: > > thank you Carlos! You did a great job figuring out this fix :) > > > > I can confirm that after applying this patch in our cluster the issue > > seems to be solved for us. Now we pass these tests with > > "OpenBLAS/0.3.1-GCC-7.3.0-2.30": > > https://github.com/eylenth/Openblas_matrix_issue > > https://github.com/xianyi/BLAS-Tester > > > > I also got a confirmation from a colleague in our user support team that > > a problem he was trying to debug with some R code is solved after this > > fix was applied. > > > > I have sent a PR with the fix upstream: > > https://github.com/easybuilders/easybuild-easyconfigs/pull/8396 > > > > In case anyone else test the workaround it would be nice if you report > > in the mailing list or in the pull request in github if it's working > > fine for you too. > > > > regards, > > Pablo > > > > > > On Tue, May 28, 2019 at 2:32 PM Carlos Fenoy <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi, > > > > After fighting a long time with this, we managed to get a solution > > that passes both the "Openblas_matrix_issue" and "BLAS_tester" test > > suites. > > > > To solve the issue we had to apply a patch and add a new build > > parameter (USE_SIMPLE_THREADED_LEVEL3=1) to OpenBLAS to make it work > > with multiple openmp threads. > > > > This is how the buildopts line looks like for us: > > > > buildopts = ' USE_SIMPLE_THREADED_LEVEL3=1 BINARY=64 USE_THREAD=1 > > USE_OPENMP=1 CC="$CC" FC="$F77" DYNAMIC_ARCH=1' > > > > And the patch, we got it from this commit on the OpenBLAS repo: > > > > https://github.com/xianyi/OpenBLAS/commit/b14f44d2adbe1ec8ede0cdf06fb8b09f3c4b6e43 > > (you > > can get the patch by adding .patch at the end of the URL) > > > > Regards, > > Carlos > > > > On Mon, May 27, 2019 at 6:15 PM Pablo Escobar Lopez > > <[email protected] <mailto:[email protected]>> > > wrote: > > > > Hi, > > > > did anyone found a working patch or workaround for the matrix > > issue when using OpenBLAS-0.3.1 ? > > > > After a lot of try&error I couldn't pass the tests in > > https://github.com/eylenth/Openblas_matrix_issue when using > > > > https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb > > . > > No matter what patches, toolchainopts or buildopts I use (and I > > have tried few different combinations) . Is anyone able to pass > > the tests using openblas-0.3.1 ? > > > > I could pass the tests using openblas-0.3.5 but upgrading my > > foss/2018b toolchain would be quite messy because I use RPATH. > > The less intrusive solution for my users would be to be able to > > patch openblas-0.3.1 somehow but I couldn't find a working > > solution. Any suggestions? > > > > regards, > > Pablo. > > > > p.s. in a related topic, IMHO unless there is a proper > > workaround I would suggest to stop providing openblas-0.3.1 with > > easybuild. Right now we are distributing a broken library > > > > > > On Tue, May 7, 2019 at 6:34 PM Mikael Öhman <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi Thomas, > > > > I can also confirm these issues. I tried rebuilding > > OpenBLAS+R after the fix in #7180, but I still saw the same > > problems. > > Very large matrix-matrix multiplications randomly gave the > > wrong result. Very large errors. The larger the matrix, the > > more frequent the errors. > > > > In the end, I compiled an intel-version (but I had to remove > > a few extensions that didn't build) and removed my Foss > > version from our installations. > > > > Perhaps it's related to hardware; I saw this on happen > > skylake servers. I haven't had time to check if this > > > > https://github.com/easybuilders/easybuild-easyconfigs/issues/8197 > > also affects 0.3.1 > > > > Best regards, Mikael > > > > > > On Tue, May 7, 2019 at 6:12 PM Thomas Eylenbosch > > <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hello____ > > > > __ __ > > > > Some of our end users reported a calculation issue with > > matrices when they are working with a foss/2018b module____ > > > > __ __ > > > > I reproduced this error with Python and R that are > > compiled with the foss/2018b toolchain, the output > > returns unexcepted results. ____ > > > > Then I reproduced this error with Python and R that are > > compiled with the foss/2016b toolchain , then it gives > > me the expected behavior. ____ > > > > __ __ > > > > You can reproduce this error with the following github > > repository: ____ > > > > https://github.com/eylenth/Openblas_matrix_issue____ > > > > __ __ > > > > I have also tried to recompile the > > OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb easyconfig file with > > “toolchainopts = {'vectorize': False}” ( cfr. > > > > https://github.com/easybuilders/easybuild-easyconfigs/issues/7180)____ > > > > But is still giving me unexpected behavior____ > > > > __ __ > > > > Can someone try to reproduce the error with the > > R/Python(foss/2018b) modules. Or can someone give me > > feedback on this?____ > > > > __ __ > > > > Thank you in advance. ____ > > > > __ __ > > > > Met vriendelijke groet / Kind regards / Beste Grüße*____* > > > > *Thomas Eylenbosch____* > > > > /Ext: Gluo N.V.//____/ > > > > __ __ > > > > BASF Agricultural Solutions Belgium NV____ > > > > Technologiepark 101____ > > > > B-9052 Ghent (Zwijnaarde)____ > > > > BELGIUM____ > > > > E-mail: *[email protected] > > <mailto:[email protected]>*____ > > > > cid:[email protected]____ > > > > BASF Agricultural Solutions Belgium NV, Registered > > Office: 9052 Gent, Belgium____ > > > > Registration: RPR Gent: 0685.756.742____ > > > > __ __ > > > > > > > > -- > > Pablo Escobar López > > Linux/HPC systems engineer > > sciCORE, University of Basel > > SIB Swiss Institute of Bioinformatics > > > > > > > > -- > > Pablo Escobar López > > Linux/HPC systems engineer > > sciCORE, University of Basel > > SIB Swiss Institute of Bioinformatics -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
