Re: [Numpy-discussion] how to use numpy-vendor
Hi Ondrej, On Tue, Aug 14, 2012 at 5:34 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote: Hi, How should one use the vendor repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. There is no information on vendor: that's a repo I set up to avoid polluting the main repo with all the binary stuff that used to be in SVN. The principle is to put binaries used to *build* numpy, but we don't put anything there for end-users. What binaries do you need to put there ? Numpy binaries are usually put on sourceforge (although I would be more than happy to have a suggestion for a better way because uploading on sourceforge is the very definition of pain). David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use numpy-vendor
On Tue, Aug 14, 2012 at 11:06 AM, David Cournapeau courn...@gmail.com wrote: Hi Ondrej, On Tue, Aug 14, 2012 at 5:34 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote: Hi, How should one use the vendor repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. There is no information on vendor: that's a repo I set up to avoid polluting the main repo with all the binary stuff that used to be in SVN. The principle is to put binaries used to *build* numpy, but we don't put anything there for end-users. What binaries do you need to put there ? Numpy binaries are usually put on sourceforge (although I would be more than happy to have a suggestion for a better way because uploading on sourceforge is the very definition of pain). I think he's asking how to use the binaries in numpy-vendor to build a release version of numpy. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use numpy-vendor
On 08/14/2012 06:34 AM, Ondřej Čertík wrote: Hi, How should one use the vendor repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me. One thing to be aware of with OpenBlas is that it is *very* tuned to the CPU at hand. As in, every CPU has hand-coded *assembly* and the makefile more or less probes for which specific CPU generation from which vendor you have and compiles and link the corresponding assembly file. So you may have to take some care that you don't compile and ship a version that breaks if you don't have SSE3 installed etc... (Unless OpenBlas has changed recently. I'm not saying I'm right, I'm saying it should be looked into.) Dag In wine, since all the binaries are .a files, the only way to check that it works is to install it and check that things like eigh() are much faster. Is there some other way? On linux using openblas as an .so library, I just do ldd and all is clear. Ondrej [1] http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use numpy-vendor
On Tue, Aug 14, 2012 at 11:22 AM, Nathaniel Smith n...@pobox.com wrote: On Tue, Aug 14, 2012 at 11:06 AM, David Cournapeau courn...@gmail.com wrote: Hi Ondrej, On Tue, Aug 14, 2012 at 5:34 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote: Hi, How should one use the vendor repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. There is no information on vendor: that's a repo I set up to avoid polluting the main repo with all the binary stuff that used to be in SVN. The principle is to put binaries used to *build* numpy, but we don't put anything there for end-users. What binaries do you need to put there ? Numpy binaries are usually put on sourceforge (although I would be more than happy to have a suggestion for a better way because uploading on sourceforge is the very definition of pain). I think he's asking how to use the binaries in numpy-vendor to build a release version of numpy. Hm, good point, I don't know why I read putting .wine stuff into vendor instead of the opposite. Anyway, the way to use the binaries is to put them in some known location, e.g. C:\local ($WINEPREFIX/drive_c/local for wine), and copy the nosse/sse2/sse3 directories in there. For example: C:\local\lib\yop\nosse C:\local\lib\yop\sse2 ... This is then referred through env by the pavement script (see https://github.com/numpy/numpy/blob/master/pavement.py#L143). Renaming yop to atlas would be a good idea, don't know why I let that non-descriptive name in there. Manually, you can just do something like ATLAS=C:\local\lib\yop\sse2 python setup.py build, but being careful about how env variables are passed between shell and wine (don't remember the details). Note that the nosse is not ATLAS, but straight netlib libs, which is why in that case you need to use BLAS=... LAPACK=... I would strongly suggest not to use openblas for this release, because of all the issues related to CPU tuning. We could certainly update a bit what we have in there, but building windows binaries is big enough of a pain, that you don't want to do everything at once I think, especially testing/building blas on windows is very time consuming. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use numpy-vendor
On Tue, Aug 14, 2012 at 6:33 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: On 08/14/2012 06:34 AM, Ondřej Čertík wrote: Hi, How should one use the vendor repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me. One thing to be aware of with OpenBlas is that it is *very* tuned to the CPU at hand. As in, every CPU has hand-coded *assembly* and the makefile more or less probes for which specific CPU generation from which vendor you have and compiles and link the corresponding assembly file. So you may have to take some care that you don't compile and ship a version that breaks if you don't have SSE3 installed etc... (Unless OpenBlas has changed recently. I'm not saying I'm right, I'm saying it should be looked into.) OpenBlas as the option to build all version and select at run time the right one. MKL do the same. But I never tested it. So I don't know how well it work. The other option would be to force an older CPU that support only sse2. ATLAS do that by default. OpenBLAS select the best one for the computer where it is being built by default. Fred ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use numpy-vendor
On Tue, Aug 14, 2012 at 6:32 AM, Frédéric Bastien no...@nouiz.org wrote: On Tue, Aug 14, 2012 at 6:33 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: On 08/14/2012 06:34 AM, Ondřej Čertík wrote: Hi, How should one use the vendor repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me. One thing to be aware of with OpenBlas is that it is *very* tuned to the CPU at hand. As in, every CPU has hand-coded *assembly* and the makefile more or less probes for which specific CPU generation from which vendor you have and compiles and link the corresponding assembly file. So you may have to take some care that you don't compile and ship a version that breaks if you don't have SSE3 installed etc... (Unless OpenBlas has changed recently. I'm not saying I'm right, I'm saying it should be looked into.) OpenBlas as the option to build all version and select at run time the right one. MKL do the same. But I never tested it. So I don't know how well it work. The other option would be to force an older CPU that support only sse2. ATLAS do that by default. OpenBLAS select the best one for the computer where it is being built by default. For the record, I don't plan to ship openblas, I was just playing with it to make sure I understand how things work. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use numpy-vendor
On Tue, Aug 14, 2012 at 3:43 AM, David Cournapeau courn...@gmail.com wrote: On Tue, Aug 14, 2012 at 11:22 AM, Nathaniel Smith n...@pobox.com wrote: On Tue, Aug 14, 2012 at 11:06 AM, David Cournapeau courn...@gmail.com wrote: Hi Ondrej, On Tue, Aug 14, 2012 at 5:34 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote: Hi, How should one use the vendor repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. There is no information on vendor: that's a repo I set up to avoid polluting the main repo with all the binary stuff that used to be in SVN. The principle is to put binaries used to *build* numpy, but we don't put anything there for end-users. What binaries do you need to put there ? Numpy binaries are usually put on sourceforge (although I would be more than happy to have a suggestion for a better way because uploading on sourceforge is the very definition of pain). I think he's asking how to use the binaries in numpy-vendor to build a release version of numpy. Yes. Hm, good point, I don't know why I read putting .wine stuff into vendor instead of the opposite. Anyway, the way to use the binaries is to put them in some known location, e.g. C:\local ($WINEPREFIX/drive_c/local for wine), and copy the nosse/sse2/sse3 directories in there. For example: C:\local\lib\yop\nosse C:\local\lib\yop\sse2 ... This is then referred through env by the pavement script (see https://github.com/numpy/numpy/blob/master/pavement.py#L143). Renaming yop to atlas would be a good idea, don't know why I let that non-descriptive name in there. I'll send a PR. Got it, thanks for your help. I'll also send a PR to the vendor repository, so that it's clear how to actually use it with NumPy. Manually, you can just do something like ATLAS=C:\local\lib\yop\sse2 python setup.py build, but being careful about how env variables are passed between shell and wine (don't remember the details). Note that Right. the nosse is not ATLAS, but straight netlib libs, which is why in that case you need to use BLAS=... LAPACK=... I would strongly suggest not to use openblas for this release, because of all the issues related to CPU tuning. We could certainly update a bit what we have in there, but building windows binaries is big enough of a pain, that you don't want to do everything at once I think, especially testing/building blas on windows is very time consuming. Absolutely, I don't plan to use nor ship openblase. My apologies for the confusion. I was just using it on linux to understand how to make numpy use it (with the ATLAS, BLAS and LAPACK env variables). Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use numpy-vendor
On Tue, Aug 14, 2012 at 7:26 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Tue, Aug 14, 2012 at 3:43 AM, David Cournapeau courn...@gmail.com wrote: [...] This is then referred through env by the pavement script (see https://github.com/numpy/numpy/blob/master/pavement.py#L143). Renaming yop to atlas would be a good idea, don't know why I let that non-descriptive name in there. I'll send a PR: https://github.com/numpy/numpy/pull/386 Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use numpy-vendor
Hi,
I've uploaded the binaries here:
https://sourceforge.net/projects/numpy/files/NumPy/1.7.0beta/
The only thing that's missing are Mac binaries, otherwise everything
else is there.
Here is the full log from the build (you have to click on View Raw
as the log is long):
https://gist.github.com/3352057
What is the best way to test, that they indeed were built with the
atlas from the vendor repository?
The relevant part of the log says:
[127.0.0.1:] out: lapack_info:
[127.0.0.1:] out: FOUND:
[127.0.0.1:] out: libraries = ['lapack']
[127.0.0.1:] out: library_dirs = ['C:\\local\\lib\\yop\\nosse']
[127.0.0.1:] out: language = f77
[127.0.0.1:] out: FOUND:
[127.0.0.1:] out: libraries = ['lapack', 'blas']
[127.0.0.1:] out: library_dirs = ['C:\\local\\lib\\yop\\nosse']
[127.0.0.1:] out: define_macros = [('NO_ATLAS_INFO', 1)]
[127.0.0.1:] out: language = f77
and
[127.0.0.1:] out: FOUND:
[127.0.0.1:] out: libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
[127.0.0.1:] out: library_dirs = ['C:\\local\\lib\\yop\\sse2']
[127.0.0.1:] out: language = f77
[127.0.0.1:] out: define_macros = [('NO_ATLAS_INFO', -1)]
and
[127.0.0.1:] out: FOUND:
[127.0.0.1:] out: libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
[127.0.0.1:] out: library_dirs = ['C:\\local\\lib\\yop\\sse3']
[127.0.0.1:] out: language = f77
[127.0.0.1:] out: define_macros = [('NO_ATLAS_INFO', -1)]
So that seems that it found it, correct?
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to use numpy-vendor
Hi, How should one use the vendor repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me. In wine, since all the binaries are .a files, the only way to check that it works is to install it and check that things like eigh() are much faster. Is there some other way? On linux using openblas as an .so library, I just do ldd and all is clear. Ondrej [1] http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
