Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On Thu, Jun 13, 2013 at 1:25 PM, Andrew Jaffe a.h.ja...@gmail.com wrote: On 11/06/2013 22:11, Chris Barker - NOAA Federal wrote: On Tue, Jun 11, 2013 at 1:28 PM, Ralf Gommers ralf.gomm...@gmail.com wrote: The binaries will still be built against python.org Python, so there shouldn't be an issue here. Same for building from source. My point was that it's nice to be able to have it build with an out of teh box wetup.py with accelerated LAPACK and all... If whoever is building binaries wants to get fancy, great. Yes, please. The current system does seem to work for at least some of us. And, if I understand the thread in the scipy mailing list, it's not actually clear that there's a bug, as opposed to incompatible fortran ABIs (which doesn't seem like a bug to me). But I guess the most important thing would be that it can be used with apple or python.org Python builds (my reading of some of the suggestions would be requiring one of homebrew/fink/macports), No we should support all pythons; python.org being the one supported by our binary installers. Ralf preferably out-of-the-box -- even if that meant restricting to prebuilt binaries. Being able to run non-obscure installers (i.e., from the main python.org and scipy.org sites) for Python + numpy + scipy + matplotlib and get optimized versions would be sufficient. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On 11/06/2013 22:11, Chris Barker - NOAA Federal wrote: On Tue, Jun 11, 2013 at 1:28 PM, Ralf Gommers ralf.gomm...@gmail.com wrote: The binaries will still be built against python.org Python, so there shouldn't be an issue here. Same for building from source. My point was that it's nice to be able to have it build with an out of teh box wetup.py with accelerated LAPACK and all... If whoever is building binaries wants to get fancy, great. Yes, please. The current system does seem to work for at least some of us. And, if I understand the thread in the scipy mailing list, it's not actually clear that there's a bug, as opposed to incompatible fortran ABIs (which doesn't seem like a bug to me). But I guess the most important thing would be that it can be used with apple or python.org Python builds (my reading of some of the suggestions would be requiring one of homebrew/fink/macports), preferably out-of-the-box -- even if that meant restricting to prebuilt binaries. Being able to run non-obscure installers (i.e., from the main python.org and scipy.org sites) for Python + numpy + scipy + matplotlib and get optimized versions would be sufficient. Andrew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
12.06.2013 00:29, Ralf Gommers kirjoitti: [clip] Sounds like a good idea. Would still make sense to move Accelerate down in the list of preferred libs, so that one can install ATLAS, MKL or OpenBLAS once and be done, instead of always having to remember these envvars. It goes like this: https://github.com/pv/numpy/tree/fortran-abicheck Interested parties on OSX should check whether they manage to build this version of Numpy with Accelerate when this is enabled, and whether Scipy's ARPACK tests still fail when building against this version of Numpy. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On Tue, Jun 11, 2013 at 11:57 PM, Pauli Virtanen p...@iki.fi wrote: 12.06.2013 00:29, Ralf Gommers kirjoitti: [clip] AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, so doing something like export FOPT=-ff2c [clip] Sounds like a good idea. Would still make sense to move Accelerate down in the list of preferred libs, so that one can install ATLAS, MKL or OpenBLAS once and be done, instead of always having to remember these envvars. Btw, it would be interesting to actually check if some combination of -ff2c in FOPT/FFLAGS/LDFLAGS/... fixes the Scipy Arpack issues. https://github.com/scipy/scipy/issues/2256#issuecomment-17028321 Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On Tue, Jun 11, 2013 at 9:22 AM, Pauli Virtanen p...@iki.fi wrote: Hi, numpy.distutils provides support for Accelerate+veclib on OSX, but does not provide Fortran compiler flags that ensure that the Fortran ABI used is compatible. As a result ASUM, SDOT, (apparently also DDOT), and other common BLAS functions return garbage when called with the wrong ABI. This breaks parts of Scipy. (There's some code to hack around this --- it worked on earlier versions of OSX, but apparently is unreliable on 10.8, demonstrating that this approach is just too brittle.) Moreover, third-party module developers who use Fortran will not be aware of this issue, so it may also break 3rd party code. As far as I see, the options are: 1. Add -ff2c (or whatever is the correct flag for Accelerate) to the Fortran flags on OSX. What is the default ABI used on homebrew ? I think we should just follow that, given that Apple cannot figure it out. 2. Drop support for Veclib/Accelerate. I think Accelerate was also incompatible with multiprocessing, which would weigh for option 2. As for multiprocessing support, the problem will happen with other libraries as well. Essentially, the way multiprocessing work (fork wo exec) is simply not supported on OS X. Olivier Grisel from scikits learn knows more about this (and has some workaround to use multiprocessing with numpy/scipy on os x). I will look into the issue with Accelerate on 10.8 (am still on 10.7 myself), but I would not mind dropping support for it if it is too much of an hassle. I will look into the bento issue that makes it fail on 64 bits python, and maybe using openblas should become the default ? David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
David Cournapeau cournape at gmail.com writes: [clip] What is the default ABI used on homebrew ? I think we should just follow that, given that Apple cannot figure it out. I think for Scipy homebrew uses the Gfortran ABI: https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile But that's probably the wrong thing to do, it doesn't work: http://trac.macports.org/ticket/36694 For Octave, they have -ff2c: https://trac.macports.org/browser/trunk/dports/math/octave/Portfile *** A third option (maybe the best one) could be to add an ABI check to numpy.distutils BLAS/LAPACK detection --- compile a small test program that checks SDOT/CDOTU/DDOT etc., and refuse to use the BLAS/LAPACK libraries if they give incorrect results. After that, we can also remove the sdot/cdotu wrappers. This approach is used by Octave. This leaves the problem of dealing with Fortran ABI to those in charge of the build environment, e.g., macports, Enthought, ..., who are also in the best position to pick the correct solution per each platform supported. AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, so doing something like export FOPT=-ff2c or export LDFLAGS=-ldotwrp -lblas works? This makes things a bit more complicated to the builder, an issue that can be solved with documentation, and keeping that up to date is easier than hardcoding stuff into numpy.distutils. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
Hi, On Tue, Jun 11, 2013 at 5:17 AM, Pauli Virtanen p...@iki.fi wrote: David Cournapeau cournape at gmail.com writes: [clip] What is the default ABI used on homebrew ? I think we should just follow that, given that Apple cannot figure it out. I think for Scipy homebrew uses the Gfortran ABI: https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile But that's probably the wrong thing to do, it doesn't work: http://trac.macports.org/ticket/36694 For Octave, they have -ff2c: https://trac.macports.org/browser/trunk/dports/math/octave/Portfile *** A third option (maybe the best one) could be to add an ABI check to numpy.distutils BLAS/LAPACK detection --- compile a small test program that checks SDOT/CDOTU/DDOT etc., and refuse to use the BLAS/LAPACK libraries if they give incorrect results. After that, we can also remove the sdot/cdotu wrappers. This approach is used by Octave. This leaves the problem of dealing with Fortran ABI to those in charge of the build environment, e.g., macports, Enthought, ..., who are also in the best position to pick the correct solution per each platform supported. AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, so doing something like export FOPT=-ff2c or export LDFLAGS=-ldotwrp -lblas works? This makes things a bit more complicated to the builder, an issue that can be solved with documentation, and keeping that up to date is easier than hardcoding stuff into numpy.distutils. What will be the performance drop for the default OSX installer version of numpy, if we drop Accelerate / veclib support? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
I think for Scipy homebrew uses the Gfortran ABI: https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile fwiw, homebrew is not macports. it's a more recent replacement that seems to be taking over gradually. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On Tue, Jun 11, 2013 at 10:22 AM, Nils Becker n.bec...@amolf.nl wrote: fwiw, homebrew is not macports. it's a more recent replacement that seems to be taking over gradually. And then there is (or was) fink Anyway, it would be really nice if numpy could work well out-of-the box with the pyton.org python (and or the Apple-supplied one) without any need fro macports, homebrew, etc. It's actually pretty cool that Apple provides veclib, and and I liked that it got used by default with a simple setup.py build. It would be nice to keep it that way. If a user (or distributor of binaries) wants to build the whole scipy stack, they'll need to figure out all this *^% anyway, changing a config in numpy to build differently would be the easiest part. And yes, there really are folks that use numpy a lot without scipy. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?
On Tue, Jun 11, 2013 at 9:44 PM, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: On Tue, Jun 11, 2013 at 10:22 AM, Nils Becker n.bec...@amolf.nl wrote: fwiw, homebrew is not macports. it's a more recent replacement that seems to be taking over gradually. And then there is (or was) fink Anyway, it would be really nice if numpy could work well out-of-the box with the pyton.org python (and or the Apple-supplied one) without any need fro macports, homebrew, etc. The binaries will still be built against python.org Python, so there shouldn't be an issue here. Same for building from source. It's actually pretty cool that Apple provides veclib, and and I liked that it got used by default with a simple setup.py build. Yeah, I still have an OS X 10.6 machine and Accelerate works great there. But they screwed it up pretty bad in 10.7, and then made it worse for 10.8. Not so cool anymore It would be nice to keep it that way. If a user (or distributor of binaries) wants to build the whole scipy stack, they'll need to figure out all this *^% anyway, changing a config in numpy to build differently would be the easiest part. And yes, there really are folks that use numpy a lot without scipy. I think we have to indeed keep it easy to build from source. Maybe providing a script to automatically fetch all dependencies and then build (like MPL does) would be a good way to go. Ralf -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On Tue, Jun 11, 2013 at 2:09 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Tue, Jun 11, 2013 at 5:17 AM, Pauli Virtanen p...@iki.fi wrote: David Cournapeau cournape at gmail.com writes: [clip] What is the default ABI used on homebrew ? I think we should just follow that, given that Apple cannot figure it out. I think for Scipy homebrew uses the Gfortran ABI: https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile But that's probably the wrong thing to do, it doesn't work: http://trac.macports.org/ticket/36694 For Octave, they have -ff2c: https://trac.macports.org/browser/trunk/dports/math/octave/Portfile *** A third option (maybe the best one) could be to add an ABI check to numpy.distutils BLAS/LAPACK detection --- compile a small test program that checks SDOT/CDOTU/DDOT etc., and refuse to use the BLAS/LAPACK libraries if they give incorrect results. After that, we can also remove the sdot/cdotu wrappers. This approach is used by Octave. This leaves the problem of dealing with Fortran ABI to those in charge of the build environment, e.g., macports, Enthought, ..., who are also in the best position to pick the correct solution per each platform supported. AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, so doing something like export FOPT=-ff2c or export LDFLAGS=-ldotwrp -lblas works? This makes things a bit more complicated to the builder, an issue that can be solved with documentation, and keeping that up to date is easier than hardcoding stuff into numpy.distutils. What will be the performance drop for the default OSX installer version of numpy, if we drop Accelerate / veclib support? Answer on scipy-dev: http://article.gmane.org/gmane.comp.python.scientific.devel/17864 Summary: it depends. If anyone knows of better benchmarks, please share. Joern Hees just wrote up how to install with OpenBLAS, if you want to know for your application you can give it a try: http://joernhees.de/blog/2013/06/08/mac-os-x-10-8-scientific-python-with-homebrew/ Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On Tue, Jun 11, 2013 at 11:17 AM, Pauli Virtanen p...@iki.fi wrote: David Cournapeau cournape at gmail.com writes: [clip] What is the default ABI used on homebrew ? I think we should just follow that, given that Apple cannot figure it out. I think for Scipy homebrew uses the Gfortran ABI: https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile But that's probably the wrong thing to do, it doesn't work: http://trac.macports.org/ticket/36694 For Octave, they have -ff2c: https://trac.macports.org/browser/trunk/dports/math/octave/Portfile *** A third option (maybe the best one) could be to add an ABI check to numpy.distutils BLAS/LAPACK detection --- compile a small test program that checks SDOT/CDOTU/DDOT etc., and refuse to use the BLAS/LAPACK libraries if they give incorrect results. After that, we can also remove the sdot/cdotu wrappers. This approach is used by Octave. This leaves the problem of dealing with Fortran ABI to those in charge of the build environment, e.g., macports, Enthought, ..., who are also in the best position to pick the correct solution per each platform supported. AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, so doing something like export FOPT=-ff2c or export LDFLAGS=-ldotwrp -lblas works? This makes things a bit more complicated to the builder, an issue that can be solved with documentation, and keeping that up to date is easier than hardcoding stuff into numpy.distutils. Sounds like a good idea. Would still make sense to move Accelerate down in the list of preferred libs, so that one can install ATLAS, MKL or OpenBLAS once and be done, instead of always having to remember these envvars. https://github.com/scipy/scipy/issues/2256#issuecomment-17028321 has a sample test. Minor issue is that it segfaults with the wrong ABI, instead of giving an incorrect result. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
On 11 Jun 2013 22:31, Ralf Gommers ralf.gomm...@gmail.com wrote: Sounds like a good idea. Would still make sense to move Accelerate down in the list of preferred libs, so that one can install ATLAS, MKL or OpenBLAS once and be done, instead of always having to remember these envvars. These days it might make sense to just make OpenBLAS the default on all platforms. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dropping support for Accelerate/veclib?
12.06.2013 00:29, Ralf Gommers kirjoitti: [clip] AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS, so doing something like export FOPT=-ff2c [clip] Sounds like a good idea. Would still make sense to move Accelerate down in the list of preferred libs, so that one can install ATLAS, MKL or OpenBLAS once and be done, instead of always having to remember these envvars. Btw, it would be interesting to actually check if some combination of -ff2c in FOPT/FFLAGS/LDFLAGS/... fixes the Scipy Arpack issues. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
