Re: [R-sig-Fedora] Changing the BLAS from openblas on a F32 box
FYI, USE_LOCKING=1 has been toggled in Fedora, so the next openblas release (0.3.9-3) will be thread-safe by default. On Wed, 27 May 2020 at 23:46, Iñaki Ucar wrote: > > On Wed, 27 May 2020 at 23:03, Gavin Simpson wrote: > > > > Thanks (again) Iñaki. > > > > There was a typo in my reply above. I should have said: I *can't* > > answer the question re USE_LOCKING=1. > > :) > > > Those other suggestions are really helpful too; I really didn't > > understand what the difference was (I'm still not clear what the > > differences are between say openblas-openmp and openblas-openmp64), > > but I did get R to pass mgcv's thread safe test with both > > openblas-openmp and blis-openmp, so I have aliased those options for > > use too. > > Basically, openblas has a number of features that can be enabled or > disabled: 64-bit integer support, threading and parallelization of > certain parts using openmp (as, e.g., data.table does). With the > combination of these features, we end up with many different flavors, > and all these are compiled and sub-packaged separately. > > Usually, you want parallelization at the top level calling a serial > version, as mgcv does. Otherwise, you may end up with an "explosion" > of threads that is counterproductive. > > -- > Iñaki Úcar -- Iñaki Úcar ___ R-SIG-Fedora mailing list R-SIG-Fedora@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
Re: [R-sig-Fedora] Changing the BLAS from openblas on a F32 box
On Wed, 27 May 2020 at 23:03, Gavin Simpson wrote: > > Thanks (again) Iñaki. > > There was a typo in my reply above. I should have said: I *can't* > answer the question re USE_LOCKING=1. :) > Those other suggestions are really helpful too; I really didn't > understand what the difference was (I'm still not clear what the > differences are between say openblas-openmp and openblas-openmp64), > but I did get R to pass mgcv's thread safe test with both > openblas-openmp and blis-openmp, so I have aliased those options for > use too. Basically, openblas has a number of features that can be enabled or disabled: 64-bit integer support, threading and parallelization of certain parts using openmp (as, e.g., data.table does). With the combination of these features, we end up with many different flavors, and all these are compiled and sub-packaged separately. Usually, you want parallelization at the top level calling a serial version, as mgcv does. Otherwise, you may end up with an "explosion" of threads that is counterproductive. -- Iñaki Úcar ___ R-SIG-Fedora mailing list R-SIG-Fedora@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
Re: [R-sig-Fedora] Changing the BLAS from openblas on a F32 box
Thanks (again) Iñaki. There was a typo in my reply above. I should have said: I *can't* answer the question re USE_LOCKING=1. Those other suggestions are really helpful too; I really didn't understand what the difference was (I'm still not clear what the differences are between say openblas-openmp and openblas-openmp64), but I did get R to pass mgcv's thread safe test with both openblas-openmp and blis-openmp, so I have aliased those options for use too. Just using blis ( /lib64/blisblas/libblas.so.3 ) was generating a segfault when running the mgcv test. Really appreciate the help! All the best G On Wed, 27 May 2020 at 14:09, Iñaki Ucar wrote: > > On Wed, 27 May 2020 at 21:40, Gavin Simpson wrote: > > > > Thanks Iñaki, that is exactly what i was looking for, esp the last > > option which I have now configured as an alias for easy remembering. > > > > I can answer the question re USE_LOCKING=1. I think that using both > > those options is required to get thread-safety even if openblas was > > compiled for single thread use. I don't know to what extent Simon has > > engaged with upstream on this etc. > > I've seen that there is a brief note about this in the project's wiki. > I agree that a sensible default in any distro would be to build > openblas-serial with USE_LOCKING=1. But I don't understand why there > is no recommendation upstream (or I didn't find it besides the note in > the wiki) and there's no idea about the performance penalty that we > incur doing so. I brought this topic to fedora-devel. > > > All I know is that using the openblas shipped with Fedora for R is > > currently a recipe for disaster for the large GAMs we're trying to > > fit. But being able to switch to atlas temporarily is a good > > alternative. > > Note that switching to openblas-openmp (libopenblaso.so) should be > thread-safe and will probably get you a better performance than Atlas. > Also, Fedora packages blis (which provides > /lib64/blisblas/libblas.so.3). It seems to be thread-safe should be > more performant than Atlas too. > > -- > Iñaki Úcar -- Gavin Simpson, PhD [he/him/his] [t] +1 306 337 8863 • Research Scientist [f] +1 306 337 2410 Institute of Environmental [tw] @ucfagls Change & Society[w] goo.gl/Zpkdem University of Regina [w] www.iecs-uregina.ca Regina, SK S4S 0A2, Canada [iD] -0002-9084-8413 • Adjunct Professor, Department of Biology, University of Regina. ___ R-SIG-Fedora mailing list R-SIG-Fedora@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
Re: [R-sig-Fedora] Changing the BLAS from openblas on a F32 box
On Wed, 27 May 2020 at 21:40, Gavin Simpson wrote: > > Thanks Iñaki, that is exactly what i was looking for, esp the last > option which I have now configured as an alias for easy remembering. > > I can answer the question re USE_LOCKING=1. I think that using both > those options is required to get thread-safety even if openblas was > compiled for single thread use. I don't know to what extent Simon has > engaged with upstream on this etc. I've seen that there is a brief note about this in the project's wiki. I agree that a sensible default in any distro would be to build openblas-serial with USE_LOCKING=1. But I don't understand why there is no recommendation upstream (or I didn't find it besides the note in the wiki) and there's no idea about the performance penalty that we incur doing so. I brought this topic to fedora-devel. > All I know is that using the openblas shipped with Fedora for R is > currently a recipe for disaster for the large GAMs we're trying to > fit. But being able to switch to atlas temporarily is a good > alternative. Note that switching to openblas-openmp (libopenblaso.so) should be thread-safe and will probably get you a better performance than Atlas. Also, Fedora packages blis (which provides /lib64/blisblas/libblas.so.3). It seems to be thread-safe should be more performant than Atlas too. -- Iñaki Úcar ___ R-SIG-Fedora mailing list R-SIG-Fedora@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
Re: [R-sig-Fedora] Changing the BLAS from openblas on a F32 box
Thanks Iñaki, that is exactly what i was looking for, esp the last option which I have now configured as an alias for easy remembering. I can answer the question re USE_LOCKING=1. I think that using both those options is required to get thread-safety even if openblas was compiled for single thread use. I don't know to what extent Simon has engaged with upstream on this etc. All I know is that using the openblas shipped with Fedora for R is currently a recipe for disaster for the large GAMs we're trying to fit. But being able to switch to atlas temporarily is a good alternative. Cheers G On Wed, 27 May 2020 at 03:20, Iñaki Ucar wrote: > > Of course, even a simpler trick is to launch R as follows: > > LD_PRELOAD=/lib64/atlas/libsatlas.so.3 R > > and then the symbols in libsatlas take precedence over libopenblas. Or > a mix between both alternatives, i.e., setting > > LD_PRELOAD=/path/to/some/link R > > and then change that link to point to openblas, atlas... Whatever > suits you best. > > Iñaki > > On Wed, 27 May 2020 at 11:00, Iñaki Ucar wrote: > > > > Hi Gavin, > > > > On Wed, 27 May 2020 at 01:15, Gavin Simpson wrote: > > > > > > Dear list, > > > > > > What is the recommended incantation on Fedora 32 to swap out the > > > openblas BLAS that the packaged (rpm) version of R-core installs for > > > ATLAS? > > > > I'm afraid there is no official mechanism in place to do that yet. > > There was a proposal [1], but it was never pushed forward due to some > > issues and lack of time. There's a simple hack you can do though. > > > > Since recently, R in Fedora no longer uses openblas-Rblas and links > > against system openblas, as you can see here: > > > > $ ldd /usr/lib64/R/bin/exec/R | grep blas > > libopenblas.so.0 => /lib64/libopenblas.so.0 (0x7f8fff849000) > > > > So one simple trick to override that is something along these lines: > > > > $ mkdir ~/blas > > $ ln -s /lib64/atlas/libsatlas.so.3 ~/blas/libopenblas.so.0 > > $ LD_LIBRARY_PATH=~/blas ldd /usr/lib64/R/bin/exec/R | grep blas > > libopenblas.so.0 => /home//blas/libopenblas.so.0 > > (0x7f78b3ea2000) > > > > As you can see, now R points to the link in my home, which in turn > > points to system's atlas. Now you can define an alias in your .bashrc > > to always prepend that override to R and Rscript. If you remove the > > link in your home, it will default to system's openblas. If you point > > it to another BLAS implementation, it will pick it up. > > > > Hope it helps. > > > > [1] https://fedoraproject.org/w/index.php?title=PackagingDrafts:BLAS_LAPACK > > > > > If anyone is interested, the issue Simon Wood reports with openblas > > > and *mgcv* is: > > > > > > Issues: > > > > > > ** openblas 0.3.x x<7 is not thread safe if itself compiled for single > > > thread > > >use and then called from multiple threads (unlike the reference BLAS, > > > say). > > >0.2.20 appears to be OK. For 0.3.x x>6 make USE_THREAD=0 USE_LOCKING=1 > > >to make openblas ensures thread safety (currently unclear if > > > USE_LOCKING > > >will be default from 0.3.7). > > > > Does this mean that single-threaded openblas should be built with > > USE_LOCKING=1 by default? Is there any upstream recommendation about > > this? > > > > -- > > Iñaki Úcar > > > > -- > Iñaki Úcar -- Gavin Simpson, PhD [he/him/his] [t] +1 306 337 8863 • Research Scientist [f] +1 306 337 2410 Institute of Environmental [tw] @ucfagls Change & Society[w] goo.gl/Zpkdem University of Regina [w] www.iecs-uregina.ca Regina, SK S4S 0A2, Canada [iD] -0002-9084-8413 • Adjunct Professor, Department of Biology, University of Regina. ___ R-SIG-Fedora mailing list R-SIG-Fedora@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
Re: [R-sig-Fedora] Changing the BLAS from openblas on a F32 box
Hi Gavin, On Wed, 27 May 2020 at 01:15, Gavin Simpson wrote: > > Dear list, > > What is the recommended incantation on Fedora 32 to swap out the > openblas BLAS that the packaged (rpm) version of R-core installs for > ATLAS? I'm afraid there is no official mechanism in place to do that yet. There was a proposal [1], but it was never pushed forward due to some issues and lack of time. There's a simple hack you can do though. Since recently, R in Fedora no longer uses openblas-Rblas and links against system openblas, as you can see here: $ ldd /usr/lib64/R/bin/exec/R | grep blas libopenblas.so.0 => /lib64/libopenblas.so.0 (0x7f8fff849000) So one simple trick to override that is something along these lines: $ mkdir ~/blas $ ln -s /lib64/atlas/libsatlas.so.3 ~/blas/libopenblas.so.0 $ LD_LIBRARY_PATH=~/blas ldd /usr/lib64/R/bin/exec/R | grep blas libopenblas.so.0 => /home//blas/libopenblas.so.0 (0x7f78b3ea2000) As you can see, now R points to the link in my home, which in turn points to system's atlas. Now you can define an alias in your .bashrc to always prepend that override to R and Rscript. If you remove the link in your home, it will default to system's openblas. If you point it to another BLAS implementation, it will pick it up. Hope it helps. [1] https://fedoraproject.org/w/index.php?title=PackagingDrafts:BLAS_LAPACK > If anyone is interested, the issue Simon Wood reports with openblas > and *mgcv* is: > > Issues: > > ** openblas 0.3.x x<7 is not thread safe if itself compiled for single thread >use and then called from multiple threads (unlike the reference BLAS, say). >0.2.20 appears to be OK. For 0.3.x x>6 make USE_THREAD=0 USE_LOCKING=1 >to make openblas ensures thread safety (currently unclear if USE_LOCKING >will be default from 0.3.7). Does this mean that single-threaded openblas should be built with USE_LOCKING=1 by default? Is there any upstream recommendation about this? -- Iñaki Úcar ___ R-SIG-Fedora mailing list R-SIG-Fedora@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
