Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, on https://bitbucket.org/carlkl/mingw-w64-for-python/downloads I uploaded 7z-archives for mingw-w64 and for OpenBLAS-0.2.10 for 32 bit and for 64 bit. To use mingw-w64 for Python = 3.3 you have to manually tweak the so called specs file - see readme.txt in the archive. Regards Carl 2014-07-28 4:32 GMT+02:00 Robert McGibbon rmcgi...@gmail.com: I forked Olivier's example project to use the same infrastructure for building conda binaries and deploying them to binstar, which might also be useful for some projects. https://github.com/rmcgibbo/python-appveyor-conda-example -Robert On Wed, Jul 9, 2014 at 3:53 PM, Robert McGibbon rmcgi...@gmail.com wrote: This is an awesome resource for tons of projects. Thanks Olivier! -Robert On Wed, Jul 9, 2014 at 7:00 AM, Olivier Grisel olivier.gri...@ensta.org wrote: Feodor updated the AppVeyor nodes to have the Windows SDK matching MSVC 2008 Express for Python 2. I have updated my sample scripts and we now have a working example of a free CI system for: Python 2 and 3 both for 32 and 64 bit architectures. https://github.com/ogrisel/python-appveyor-demo Best, -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
2014-07-28 15:25 GMT+02:00 Carl Kleffner cmkleff...@gmail.com: Hi, on https://bitbucket.org/carlkl/mingw-w64-for-python/downloads I uploaded 7z-archives for mingw-w64 and for OpenBLAS-0.2.10 for 32 bit and for 64 bit. To use mingw-w64 for Python = 3.3 you have to manually tweak the so called specs file - see readme.txt in the archive. Have the patches to build numpy and scipy with mingw-w64 been merged in the master branches of those projects? -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
I had to move my development enviroment on different windows box recently (stilll in progress). On this box I don't have full access unfortunately. The patch for scipy build was merged into scipy master some time ago, see https://github.com/scipy/scipy/pull/3484 . I have some additional patches for scipy.test. The pull request for numpy build has not yet been made for the reasons I mentioned. Cheers, Carl 2014-07-28 16:46 GMT+02:00 Olivier Grisel olivier.gri...@ensta.org: 2014-07-28 15:25 GMT+02:00 Carl Kleffner cmkleff...@gmail.com: Hi, on https://bitbucket.org/carlkl/mingw-w64-for-python/downloads I uploaded 7z-archives for mingw-w64 and for OpenBLAS-0.2.10 for 32 bit and for 64 bit. To use mingw-w64 for Python = 3.3 you have to manually tweak the so called specs file - see readme.txt in the archive. Have the patches to build numpy and scipy with mingw-w64 been merged in the master branches of those projects? -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
I forked Olivier's example project to use the same infrastructure for building conda binaries and deploying them to binstar, which might also be useful for some projects. https://github.com/rmcgibbo/python-appveyor-conda-example -Robert On Wed, Jul 9, 2014 at 3:53 PM, Robert McGibbon rmcgi...@gmail.com wrote: This is an awesome resource for tons of projects. Thanks Olivier! -Robert On Wed, Jul 9, 2014 at 7:00 AM, Olivier Grisel olivier.gri...@ensta.org wrote: Feodor updated the AppVeyor nodes to have the Windows SDK matching MSVC 2008 Express for Python 2. I have updated my sample scripts and we now have a working example of a free CI system for: Python 2 and 3 both for 32 and 64 bit architectures. https://github.com/ogrisel/python-appveyor-demo Best, -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
2014-07-10 0:53 GMT+02:00 Robert McGibbon rmcgi...@gmail.com: This is an awesome resource for tons of projects. Thanks. FYI here is the PR for sklearn to use AppVeyor CI: https://github.com/scikit-learn/scikit-learn/pull/3363 It's slightly different from the minimalistic sample I wrote for python-appveyor-demo in the sense that for sklearn I decided to actually install the generated wheel package and run the tests on the resulting installed library rather than on the project source folder. -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
This is an awesome resource for tons of projects. Thanks Olivier! -Robert On Wed, Jul 9, 2014 at 7:00 AM, Olivier Grisel olivier.gri...@ensta.org wrote: Feodor updated the AppVeyor nodes to have the Windows SDK matching MSVC 2008 Express for Python 2. I have updated my sample scripts and we now have a working example of a free CI system for: Python 2 and 3 both for 32 and 64 bit architectures. https://github.com/ogrisel/python-appveyor-demo Best, -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Feodor updated the AppVeyor nodes to have the Windows SDK matching MSVC 2008 Express for Python 2. I have updated my sample scripts and we now have a working example of a free CI system for: Python 2 and 3 both for 32 and 64 bit architectures. https://github.com/ogrisel/python-appveyor-demo Best, -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi! I gave appveyor a try this WE so as to build a minimalistic Python 3 project with a Cython extension. It works both with 32 and 64 bit MSVC++ and can generate wheel packages. See: https://github.com/ogrisel/python-appveyor-demo However 2008 is not (yet) installed so it cannot be used for Python 2.7. The Feodor Fitsner seems to be open to install older versions of MSVC++ on the worker VM image so this might be possible in the future. Let's see. Off-course for numpy / scipy this does not solve the fortran compiler issue, so Carl's static mingw-w64 toolchain still looks like a very promising solution (and could probably be run on the appveyor infra as well). Best, -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On 03.07.2014 05:56, Sturla Molden wrote: On 02/07/14 19:55, Chris Barker wrote: Indeed -- the default (i.e what you get with pip install numpy) should be SSE2 -- I:d much rather have a few folks with old hardware have to go through some hoops that n have most people get something that is much slower than MATLAB. I think we should use SSE3 as default. It is already ten years old. Most users (99.999 %) who want binary wheels have an SSE3 capable CPU. while true that pretty much all cpus currently around have it there is no technical requirement for even new cpus to have SSE3. Compared to SSE2 you do not have to implement it to sell a compatible 64 bit cpu. Not even the new x32 ABI requires it. In practice I think we could easily get away with using SSE3 as default but I still would like to see if it makes any performance difference in benchmarks. In my experience (which is exclusively on pre-haswell machines) the horizontal operations it offers tend to be slower than other solutions. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Thu, Jul 3, 2014 at 4:56 AM, Sturla Molden sturla.mol...@gmail.com wrote: On 02/07/14 19:55, Chris Barker wrote: Indeed -- the default (i.e what you get with pip install numpy) should be SSE2 -- I:d much rather have a few folks with old hardware have to go through some hoops that n have most people get something that is much slower than MATLAB. I think we should use SSE3 as default. It is already ten years old. Most users (99.999 %) who want binary wheels have an SSE3 capable CPU. The 99% for SSE2 comes from the Firefox crash reports, where the large majority are for very recent Firefox downloads. If you can identify SSE3 machines from the reported CPU string (as the Firefox people did for SSE2), please do have a look a see if you can get a count for SSE3 in the Firefox crash reports; if it's close to 99% that would make a strong argument: https://github.com/numpy/numpy/wiki/Windows-versions#sse--sse2 https://gist.github.com/matthew-brett/9cb5274f7451a3eb8fc0 Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
I guess this one's mainly for Carl: On Thu, Jul 3, 2014 at 11:06 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Jul 3, 2014 at 4:56 AM, Sturla Molden sturla.mol...@gmail.com wrote: On 02/07/14 19:55, Chris Barker wrote: Indeed -- the default (i.e what you get with pip install numpy) should be SSE2 -- I:d much rather have a few folks with old hardware have to go through some hoops that n have most people get something that is much slower than MATLAB. I think we should use SSE3 as default. It is already ten years old. Most users (99.999 %) who want binary wheels have an SSE3 capable CPU. The 99% for SSE2 comes from the Firefox crash reports, where the large majority are for very recent Firefox downloads. If you can identify SSE3 machines from the reported CPU string (as the Firefox people did for SSE2), please do have a look a see if you can get a count for SSE3 in the Firefox crash reports; if it's close to 99% that would make a strong argument: https://github.com/numpy/numpy/wiki/Windows-versions#sse--sse2 https://gist.github.com/matthew-brett/9cb5274f7451a3eb8fc0 Jonathan Helmus recently pointed out https://ci.appveyor.com in a discussion on the scikit-image mailing list. The scikit-image team are trying to get builds and tests working there. The configuration file allows arbitrary cmd and powershell commands executed in a clean Windows virtual machine. Do you think it would be possible to get the wheel builds working on something like that? That would be a big step forward, just because the current procedure is rather fiddly, even if not very difficult. Any news on the pull request to numpy? Waiting eagerly :) Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi Matthew, I can make it in the late evening (MEZ timezone), so you have to wait a bit ... I also will try to create new numpy/scipy wheels. I now have the latest OpenBLAS version ready. Olivier gaves me access to rackspace. I wil try it out on the weekend. Regards Carl 2014-07-03 12:46 GMT+02:00 Matthew Brett matthew.br...@gmail.com: I guess this one's mainly for Carl: On Thu, Jul 3, 2014 at 11:06 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Jul 3, 2014 at 4:56 AM, Sturla Molden sturla.mol...@gmail.com wrote: On 02/07/14 19:55, Chris Barker wrote: Indeed -- the default (i.e what you get with pip install numpy) should be SSE2 -- I:d much rather have a few folks with old hardware have to go through some hoops that n have most people get something that is much slower than MATLAB. I think we should use SSE3 as default. It is already ten years old. Most users (99.999 %) who want binary wheels have an SSE3 capable CPU. The 99% for SSE2 comes from the Firefox crash reports, where the large majority are for very recent Firefox downloads. If you can identify SSE3 machines from the reported CPU string (as the Firefox people did for SSE2), please do have a look a see if you can get a count for SSE3 in the Firefox crash reports; if it's close to 99% that would make a strong argument: https://github.com/numpy/numpy/wiki/Windows-versions#sse--sse2 https://gist.github.com/matthew-brett/9cb5274f7451a3eb8fc0 Jonathan Helmus recently pointed out https://ci.appveyor.com in a discussion on the scikit-image mailing list. The scikit-image team are trying to get builds and tests working there. The configuration file allows arbitrary cmd and powershell commands executed in a clean Windows virtual machine. Do you think it would be possible to get the wheel builds working on something like that? That would be a big step forward, just because the current procedure is rather fiddly, even if not very difficult. Any news on the pull request to numpy? Waiting eagerly :) Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Thu, Jul 3, 2014 at 12:51 PM, Carl Kleffner cmkleff...@gmail.com wrote: Hi Matthew, I can make it in the late evening (MEZ timezone), so you have to wait a bit ... I also will try to create new numpy/scipy wheels. I now have the latest OpenBLAS version ready. Olivier gaves me access to rackspace. I wil try it out on the weekend. Great - thanks a lot, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi Matthew and Ralf, Has anyone managed to build working whl packages for numpy and scipy on win32 using the static mingw-w64 toolchain? -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi all, I do regulary builds for python-2.7. Due to my limited resources I didn't build for 3.3 or 3.4 right now. I didn't updated my toolchhain from february, but I do regulary builds of OpenBLAS. OpenBLAS is under heavy development right now, thanks to Werner Saar, see: https://github.com/wernsaar/OpenBLAS . A lot of bugs have been canceled out at the cost of performance, see the kernel TODO list: https://github.com/xianyi/OpenBLAS/wiki/Fixed-optimized-kernels-To-do-List . Many bugs related to Windows have been corrected. A very weird bug i.e.: https://github.com/xianyi/OpenBLAS/issues/394 and https://github.com/JuliaLang/julia/issues/5574 . I got the impression, that the Julia community (and maybe the R and octave community) is very interested getting towards a stable Windows OpenBLAS. OpenBLAS is the only free OSS optimized BLAS/Lapack solution maintained for Windows today. Atlas seems not to be maintained for Windows anymore (is this true Matthew?) somewhat older test wheels for python-2.7 can be downloaded here: see: http://figshare.com/articles/search?q=numpyquick=1x=0y=0 (2014-06-10) numpy and scipy wheels for py-2.7 The scipy test suite (amd64) emits segfaults with multithreaded OpenBLAS, but is stable with single thread (see the log files). I didn't dig into this further. Win32 works with MT OpenBLAS, but has some test failures with atan2 and hypot. The is more or less the status today. I can upload new wheels linked against a recent OpenBLAS, maybe tomorrow on Binstar. Regards, Carl 2014-07-02 9:24 GMT+02:00 Olivier Grisel olivier.gri...@ensta.org: Hi Matthew and Ralf, Has anyone managed to build working whl packages for numpy and scipy on win32 using the static mingw-w64 toolchain? -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Wed, Jul 2, 2014 at 10:36 AM, Carl Kleffner cmkleff...@gmail.com wrote: Hi all, I do regulary builds for python-2.7. Due to my limited resources I didn't build for 3.3 or 3.4 right now. I didn't updated my toolchhain from february, but I do regulary builds of OpenBLAS. OpenBLAS is under heavy development right now, thanks to Werner Saar, see: https://github.com/wernsaar/OpenBLAS . A lot of bugs have been canceled out at the cost of performance, see the kernel TODO list: https://github.com/xianyi/OpenBLAS/wiki/Fixed-optimized-kernels-To-do-List . Many bugs related to Windows have been corrected. A very weird bug i.e.: https://github.com/xianyi/OpenBLAS/issues/394 and https://github.com/JuliaLang/julia/issues/5574 . I got the impression, that the Julia community (and maybe the R and octave community) is very interested getting towards a stable Windows OpenBLAS. OpenBLAS is the only free OSS optimized BLAS/Lapack solution maintained for Windows today. Atlas seems not to be maintained for Windows anymore (is this true Matthew?) No, it's not true, but it's not really false either. Clint Whaley is the ATLAS maintainer and his interests are firmly in high-performance-computing so he is much more interested in exotic new chips than in Windows. But, he does aim to make the latest stable release buildable on Windows, and he's helped me do that for the latest stable, with some hope he'll continue to work on the 64-bit Windows kernels which are hobbled at the moment because of differences in the Windows / other OS 64-bit ABI. Builds here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/ somewhat older test wheels for python-2.7 can be downloaded here: see: http://figshare.com/articles/search?q=numpyquick=1x=0y=0 (2014-06-10) numpy and scipy wheels for py-2.7 The scipy test suite (amd64) emits segfaults with multithreaded OpenBLAS, but is stable with single thread (see the log files). I didn't dig into this further. Win32 works with MT OpenBLAS, but has some test failures with atan2 and hypot. The is more or less the status today. I can upload new wheels linked against a recent OpenBLAS, maybe tomorrow on Binstar. I built some 64-bit wheels against Carl's toolchain and the ATLAS above, I think they don't have any threading issues, but the scipy wheel fails one scipy test due to some very small precision differences in the mingw runtime. I think we agreed this failure wasn't important. https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Wed, Jul 2, 2014 at 11:29 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Jul 2, 2014 at 10:36 AM, Carl Kleffner cmkleff...@gmail.com wrote: Hi all, I do regulary builds for python-2.7. Due to my limited resources I didn't build for 3.3 or 3.4 right now. I didn't updated my toolchhain from february, but I do regulary builds of OpenBLAS. OpenBLAS is under heavy development right now, thanks to Werner Saar, see: https://github.com/wernsaar/OpenBLAS . A lot of bugs have been canceled out at the cost of performance, see the kernel TODO list: https://github.com/xianyi/OpenBLAS/wiki/Fixed-optimized-kernels-To-do-List . Many bugs related to Windows have been corrected. A very weird bug i.e.: https://github.com/xianyi/OpenBLAS/issues/394 and https://github.com/JuliaLang/julia/issues/5574 . I got the impression, that the Julia community (and maybe the R and octave community) is very interested getting towards a stable Windows OpenBLAS. OpenBLAS is the only free OSS optimized BLAS/Lapack solution maintained for Windows today. Atlas seems not to be maintained for Windows anymore (is this true Matthew?) No, it's not true, but it's not really false either. Clint Whaley is the ATLAS maintainer and his interests are firmly in high-performance-computing so he is much more interested in exotic new chips than in Windows. But, he does aim to make the latest stable release buildable on Windows, and he's helped me do that for the latest stable, with some hope he'll continue to work on the 64-bit Windows kernels which are hobbled at the moment because of differences in the Windows / other OS 64-bit ABI. Builds here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/ somewhat older test wheels for python-2.7 can be downloaded here: see: http://figshare.com/articles/search?q=numpyquick=1x=0y=0 (2014-06-10) numpy and scipy wheels for py-2.7 The scipy test suite (amd64) emits segfaults with multithreaded OpenBLAS, but is stable with single thread (see the log files). I didn't dig into this further. Win32 works with MT OpenBLAS, but has some test failures with atan2 and hypot. The is more or less the status today. I can upload new wheels linked against a recent OpenBLAS, maybe tomorrow on Binstar. I built some 64-bit wheels against Carl's toolchain and the ATLAS above, I think they don't have any threading issues, but the scipy wheel fails one scipy test due to some very small precision differences in the mingw runtime. I think we agreed this failure wasn't important. https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl Sorry - I wasn't paying attention - you asked about 32-bit wheels. Honestly, using the same toolchain, they wouldn't be at all hard to build. One issue is that the ATLAS builds depend on SSE2. That isn't an issue for 64 bit builds because the 64-bit ABI requires SSE2, but it is an issue for 32-bit where we have no such guarantee. It looks like 99% of Windows users do have SSE2 though [1]. So I think what is required is * Build the wheels for 32-bit (easy) * Patch the wheels to check and give helpful error in absence of SSE2 (fairly easy) * Get agreement these should go up on pypi and be maintained (feedback anyone?) Cheers, Matthew [1] https://github.com/numpy/numpy/wiki/Windows-versions#sse--sse2 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, The mingw-w64 based wheels (Atlas and openBLAS) are based on a patched numpy version, that hasn't been published as numpy pull for revision until now (my failure). I could try to do this tomorrow in the evening. Another important point is, that the toolchain, that is capable to compile numpy/scipy was adapted to allow for MSVC / mingw runtime compatibility and does not create any gcc/mingw runtime dependency anymore. OpenBLAS has one advantage over Atlas: numpy/scipy are linked dynamically against OpenBLAS. Statically linked BLAS like MKL or ATLAS creates huge python extensions and have considerable higher memory consumption compared to dynamically linkage. On the other hand correctness is more important, so ATLAS has to be preferred now. Users with non SEE processors could be provided with wheels distributed on binstar. Regards Carl 2014-07-02 12:37 GMT+02:00 Matthew Brett matthew.br...@gmail.com: Hi, On Wed, Jul 2, 2014 at 11:29 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Jul 2, 2014 at 10:36 AM, Carl Kleffner cmkleff...@gmail.com wrote: Hi all, I do regulary builds for python-2.7. Due to my limited resources I didn't build for 3.3 or 3.4 right now. I didn't updated my toolchhain from february, but I do regulary builds of OpenBLAS. OpenBLAS is under heavy development right now, thanks to Werner Saar, see: https://github.com/wernsaar/OpenBLAS . A lot of bugs have been canceled out at the cost of performance, see the kernel TODO list: https://github.com/xianyi/OpenBLAS/wiki/Fixed-optimized-kernels-To-do-List . Many bugs related to Windows have been corrected. A very weird bug i.e.: https://github.com/xianyi/OpenBLAS/issues/394 and https://github.com/JuliaLang/julia/issues/5574 . I got the impression, that the Julia community (and maybe the R and octave community) is very interested getting towards a stable Windows OpenBLAS. OpenBLAS is the only free OSS optimized BLAS/Lapack solution maintained for Windows today. Atlas seems not to be maintained for Windows anymore (is this true Matthew?) No, it's not true, but it's not really false either. Clint Whaley is the ATLAS maintainer and his interests are firmly in high-performance-computing so he is much more interested in exotic new chips than in Windows. But, he does aim to make the latest stable release buildable on Windows, and he's helped me do that for the latest stable, with some hope he'll continue to work on the 64-bit Windows kernels which are hobbled at the moment because of differences in the Windows / other OS 64-bit ABI. Builds here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/ somewhat older test wheels for python-2.7 can be downloaded here: see: http://figshare.com/articles/search?q=numpyquick=1x=0y=0 (2014-06-10) numpy and scipy wheels for py-2.7 The scipy test suite (amd64) emits segfaults with multithreaded OpenBLAS, but is stable with single thread (see the log files). I didn't dig into this further. Win32 works with MT OpenBLAS, but has some test failures with atan2 and hypot. The is more or less the status today. I can upload new wheels linked against a recent OpenBLAS, maybe tomorrow on Binstar. I built some 64-bit wheels against Carl's toolchain and the ATLAS above, I think they don't have any threading issues, but the scipy wheel fails one scipy test due to some very small precision differences in the mingw runtime. I think we agreed this failure wasn't important. https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl Sorry - I wasn't paying attention - you asked about 32-bit wheels. Honestly, using the same toolchain, they wouldn't be at all hard to build. One issue is that the ATLAS builds depend on SSE2. That isn't an issue for 64 bit builds because the 64-bit ABI requires SSE2, but it is an issue for 32-bit where we have no such guarantee. It looks like 99% of Windows users do have SSE2 though [1]. So I think what is required is * Build the wheels for 32-bit (easy) * Patch the wheels to check and give helpful error in absence of SSE2 (fairly easy) * Get agreement these should go up on pypi and be maintained (feedback anyone?) Cheers, Matthew [1] https://github.com/numpy/numpy/wiki/Windows-versions#sse--sse2 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Wed, Jul 2, 2014 at 12:18 PM, Carl Kleffner cmkleff...@gmail.com wrote: Hi, The mingw-w64 based wheels (Atlas and openBLAS) are based on a patched numpy version, that hasn't been published as numpy pull for revision until now (my failure). I could try to do this tomorrow in the evening. That would be really good. I'll try and help with review if I can. Another important point is, that the toolchain, that is capable to compile numpy/scipy was adapted to allow for MSVC / mingw runtime compatibility and does not create any gcc/mingw runtime dependency anymore. OpenBLAS has one advantage over Atlas: numpy/scipy are linked dynamically against OpenBLAS. Statically linked BLAS like MKL or ATLAS creates huge python extensions and have considerable higher memory consumption compared to dynamically linkage. On the other hand correctness is more important, so ATLAS has to be preferred now. Do you have any index of what the memory cost is? If it's in the order of 20M presumably that won't have much practical impact? Users with non SEE processors could be provided with wheels distributed on binstar. The last plan we seemed to have was to continue making the 'superpack' exe installers which contain no-SSE, SSE2 and SSE3 builds where the installer selects which one to install at runtime. The warning from the wheel would point to these installers as the backup option. If we did want to produce alternative wheels, I guess a specific static https directory would be easiest; otherwise the user would get the odd effect that they'd get a hobbled wheel by default when installing from binstar (assuming they did in fact have SSE2). I mean, this pip install -f https://somewhere.org/no_sse_wheels --no-index numpy seems to make more sense as an alternative install command for non-SSE, than this: pip install -i http://binstar.org numpy because in the former case, you can see what is special about the command. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi Carl, All the items you suggest would be very appreciated. Don't hesitate to ping me if you need me to test new packages. Also the sklearn project has a free Rackspace Cloud account that Matthew is already using to make travis upload OSX wheels for the master branch of various scipy stack projects. Rackspace cloud can also be used to start windows VMs if needed. Please tell me if you want a some user credentials and API key. Myself I use the Rackspace Cloud account to build sklearn wheels following those instructions: https://github.com/scikit-learn/scikit-learn/wiki/How-to-make-a-release#building-windows-binary-packages We are using msvc express (but only for 32bit Python) right now. I have yet to try to build sklearn with your mingw-w64 static toolchain. Rackspace granted us $2000 worth of cloud resource per month (e.g. bandwith and VM time) so there is plenty of resource left to help with upstream projects such as numpy and scipy. Best, -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, personally I don't have a preference of Binstar over somewhere.org. More important is that one has to agree where to find the binaries. Binstar has the concept of channels and allow wheels. So one could provide a channel for NOSSE and more channels for other specialized builds: ATLAS/OpenBLAS/RefBLAS, SSE4/AVX and so on. A generic binary should be build with generic optimizing GCC switches and SSE2 per default. I propose to provide generic binaries for PYPI instead of superbinaries. and specialized binaries on Binstar or somewhere else. Just thinking two or three steps ahead. Regards Carl 2014-07-02 13:35 GMT+02:00 Matthew Brett matthew.br...@gmail.com: Hi, On Wed, Jul 2, 2014 at 12:18 PM, Carl Kleffner cmkleff...@gmail.com wrote: Hi, The mingw-w64 based wheels (Atlas and openBLAS) are based on a patched numpy version, that hasn't been published as numpy pull for revision until now (my failure). I could try to do this tomorrow in the evening. That would be really good. I'll try and help with review if I can. Another important point is, that the toolchain, that is capable to compile numpy/scipy was adapted to allow for MSVC / mingw runtime compatibility and does not create any gcc/mingw runtime dependency anymore. OpenBLAS has one advantage over Atlas: numpy/scipy are linked dynamically against OpenBLAS. Statically linked BLAS like MKL or ATLAS creates huge python extensions and have considerable higher memory consumption compared to dynamically linkage. On the other hand correctness is more important, so ATLAS has to be preferred now. Do you have any index of what the memory cost is? If it's in the order of 20M presumably that won't have much practical impact? Users with non SEE processors could be provided with wheels distributed on binstar. The last plan we seemed to have was to continue making the 'superpack' exe installers which contain no-SSE, SSE2 and SSE3 builds where the installer selects which one to install at runtime. The warning from the wheel would point to these installers as the backup option. If we did want to produce alternative wheels, I guess a specific static https directory would be easiest; otherwise the user would get the odd effect that they'd get a hobbled wheel by default when installing from binstar (assuming they did in fact have SSE2). I mean, this pip install -f https://somewhere.org/no_sse_wheels --no-index numpy seems to make more sense as an alternative install command for non-SSE, than this: pip install -i http://binstar.org numpy because in the former case, you can see what is special about the command. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Wed, Jul 2, 2014 at 2:24 PM, Carl Kleffner cmkleff...@gmail.com wrote: Hi, personally I don't have a preference of Binstar over somewhere.org. More important is that one has to agree where to find the binaries. Binstar has the concept of channels and allow wheels. So one could provide a channel for NOSSE and more channels for other specialized builds: ATLAS/OpenBLAS/RefBLAS, SSE4/AVX and so on. Having a noSSE channel would make sense. A generic binary should be build with generic optimizing GCC switches and SSE2 per default. I propose to provide generic binaries for PYPI instead of superbinaries. and specialized binaries on Binstar or somewhere else. The exe superbinary installers can also go on pypi without causing confusion to pip at least, but it would be good to have wheels as well. Just thinking two or three steps ahead. It's good to have a plan :) Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Wed, Jul 2, 2014 at 3:37 AM, Matthew Brett matthew.br...@gmail.com wrote: It looks like 99% of Windows users do have SSE2 though [1]. So I think what is required is * Build the wheels for 32-bit (easy) * Patch the wheels to check and give helpful error in absence of SSE2 (fairly easy) * Get agreement these should go up on pypi and be maintained (feedback anyone?) +Inf It would benefit the community a LOT to have binary wheels up on PyPi, and the very small number of failures due to old hardware will be no big deal, as long as the users get a meaningful message, rather than a hard crash. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Wed, Jul 2, 2014 at 6:36 AM, Matthew Brett matthew.br...@gmail.com wrote: Having a noSSE channel would make sense. Indeed -- the default (i.e what you get with pip install numpy) should be SSE2 -- I:d much rather have a few folks with old hardware have to go through some hoops that n have most people get something that is much slower than MATLAB. The exe superbinary installers can also go on pypi without causing confusion to pip at least, but it would be good to have wheels as well. it doesn't hurt to have them, but we really need to get Windows away from the exe installers into the pip / virtualenv / etc world. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On 02/07/14 19:55, Chris Barker wrote: Indeed -- the default (i.e what you get with pip install numpy) should be SSE2 -- I:d much rather have a few folks with old hardware have to go through some hoops that n have most people get something that is much slower than MATLAB. I think we should use SSE3 as default. It is already ten years old. Most users (99.999 %) who want binary wheels have an SSE3 capable CPU. According to Wikipedia: AMD: Athlon 64 (since Venice Stepping E3 and San Diego Stepping E4) Athlon 64 X2 Athlon 64 FX (since San Diego Stepping E4) Opteron (since Stepping E4) Sempron (since Palermo. Stepping E3) Phenom Phenom II Athlon II Turion 64 Turion 64 X2 Turion X2 Turion X2 Ultra Turion II X2 Mobile Turion II X2 Ultra APU FX Series Intel: Celeron D Celeron (starting with Core microarchitecture) Pentium 4 (since Prescott) Pentium D Pentium Extreme Edition (but NOT Pentium 4 Extreme Edition) Pentium Dual-Core Pentium (starting with Core microarchitecture) Core Xeon (since Nocona) Atom If you have Pentium II, you can build your own NumPy... Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Fri, May 23, 2014 at 2:41 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, May 9, 2014 at 4:06 AM, David Cournapeau courn...@gmail.com wrote: On Fri, May 9, 2014 at 11:49 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 09.05.2014 12:42, David Cournapeau wrote: On Fri, May 9, 2014 at 1:51 AM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com wrote: On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com mailto:cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Fine by me (not that you need my approval !). How easy is it to build ATLAS targetting a specific CPU these days ? I think we need to at least support nosse and sse2 and above. I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I think Clint will have some time to help out next week. I did some analysis of SSE2 prevalence here: https://github.com/numpy/numpy/wiki/Window-versions Firefox crash reports now have about 1 percent of machines without SSE2. I suspect that people running new installs of numpy will have slightly better machines on average than Firefox users, but it's only a guess. I wonder if we could add a CPU check on numpy import to give a polite 'install from the exe' message for people without SSE2. We could, although you unfortunately can't do it easily from ctypes only (as you need some ASM). I can take a quick look at a simple cython extension that could be imported before anything else, and would raise an ImportError if the wrong arch is detected. assuming mingw is new enough #ifdef __SSE2___ raise_if(!__builtin_cpu_supports(sse)) #endof We need to support it for VS as well, but it looks like win32 API has a function to do it: http://msdn.microsoft.com/en-us/library/ms724482%28VS.85%29.aspx Makes it even easier. Nice. So all we would need is something like: try: from ctypes import windll, wintypes except (ImportError, ValueError): pass else: has_feature = windll.kernel32.IsProcessorFeaturePresent has_feature.argtypes = [wintypes.DWORD] if not has_feature(10): msg = (This version of numpy needs a CPU capable of SSE2, but Windows says - not so.\n, Please reinstall numpy using a superpack installer) raise RuntimeError(msg) At the top of numpy/__init__.py What would be the best way of including that code in the 32-bit wheel? (The 64-bit wheel can depend on SSE2). Maybe write a separate file `_check_win32_sse2.py.in`, and ensure that when you generate `_check_win32_sse2.py` from setup.py you only end up with the above code when you go through the if len(sys.argv) = 2 and sys.argv[1] == 'bdist_wheel': branch. Ralf Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Fri, May 9, 2014 at 4:06 AM, David Cournapeau courn...@gmail.com wrote: On Fri, May 9, 2014 at 11:49 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 09.05.2014 12:42, David Cournapeau wrote: On Fri, May 9, 2014 at 1:51 AM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com wrote: On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com mailto:cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Fine by me (not that you need my approval !). How easy is it to build ATLAS targetting a specific CPU these days ? I think we need to at least support nosse and sse2 and above. I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I think Clint will have some time to help out next week. I did some analysis of SSE2 prevalence here: https://github.com/numpy/numpy/wiki/Window-versions Firefox crash reports now have about 1 percent of machines without SSE2. I suspect that people running new installs of numpy will have slightly better machines on average than Firefox users, but it's only a guess. I wonder if we could add a CPU check on numpy import to give a polite 'install from the exe' message for people without SSE2. We could, although you unfortunately can't do it easily from ctypes only (as you need some ASM). I can take a quick look at a simple cython extension that could be imported before anything else, and would raise an ImportError if the wrong arch is detected. assuming mingw is new enough #ifdef __SSE2___ raise_if(!__builtin_cpu_supports(sse)) #endof We need to support it for VS as well, but it looks like win32 API has a function to do it: http://msdn.microsoft.com/en-us/library/ms724482%28VS.85%29.aspx Makes it even easier. Nice. So all we would need is something like: try: from ctypes import windll, wintypes except (ImportError, ValueError): pass else: has_feature = windll.kernel32.IsProcessorFeaturePresent has_feature.argtypes = [wintypes.DWORD] if not has_feature(10): msg = (This version of numpy needs a CPU capable of SSE2, but Windows says - not so.\n, Please reinstall numpy using a superpack installer) raise RuntimeError(msg) At the top of numpy/__init__.py What would be the best way of including that code in the 32-bit wheel? (The 64-bit wheel can depend on SSE2). Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi,
On Thu, May 8, 2014 at 5:51 PM, Matthew Brett matthew.br...@gmail.com wrote:
Hi,
On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau courn...@gmail.com wrote:
On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.com
wrote:
Aha,
On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com
wrote:
Hi,
On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com
wrote:
A possible option is to install the toolchain inside site-packages and
to
deploy it as PYPI wheel or wininst packages. The PATH to the toolchain
could
be extended during import of the package. But I have no idea, whats the
best
strategy to additionaly install ATLAS or other third party libraries.
Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the
devkit package. It sounds like OpenBLAS will be much easier to build,
so we could start with ATLAS binaries as a default, expecting OpenBLAS
to be built more often with the toolchain. I think that's how numpy
binary installers are built at the moment - using old binary builds of
ATLAS.
I'm happy to provide the builds of ATLAS - e.g. here:
https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds
I just found the official numpy binary builds of ATLAS:
https://github.com/numpy/vendor/tree/master/binaries
But - they are from an old version of ATLAS / Lapack, and only for 32-bit.
David - what say we update these to latest ATLAS stable?
Fine by me (not that you need my approval !).
How easy is it to build ATLAS targetting a specific CPU these days ? I think
we need to at least support nosse and sse2 and above.
I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I
think Clint will have some time to help out next week.
Clint spent an hour on the phone working through the 32-bit build.
There was a nasty gcc bug revealed by some oddness to the input flags.
Fixed now:
https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/
Configure flags needed for 32-bit:
config_opts=-b 32 -Si archdef 0 -A 13 -V 384 \
--with-netlib-lapack-tarfile=${lapack_tarfile} \
-Fa al '-mincoming-stack-boundary=2 -mfpmath=sse -msse2'
For 64-bit:
config_opts=-b 64 -V 384 --with-netlib-lapack-tarfile=${lapack_tarfile}
Cheers,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Fri, May 9, 2014 at 1:51 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau courn...@gmail.com wrote: On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.com wrote: Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Fine by me (not that you need my approval !). How easy is it to build ATLAS targetting a specific CPU these days ? I think we need to at least support nosse and sse2 and above. I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I think Clint will have some time to help out next week. I did some analysis of SSE2 prevalence here: https://github.com/numpy/numpy/wiki/Window-versions Firefox crash reports now have about 1 percent of machines without SSE2. I suspect that people running new installs of numpy will have slightly better machines on average than Firefox users, but it's only a guess. I wonder if we could add a CPU check on numpy import to give a polite 'install from the exe' message for people without SSE2. We could, although you unfortunately can't do it easily from ctypes only (as you need some ASM). I can take a quick look at a simple cython extension that could be imported before anything else, and would raise an ImportError if the wrong arch is detected. David Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Fri, May 9, 2014 at 11:49 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 09.05.2014 12:42, David Cournapeau wrote: On Fri, May 9, 2014 at 1:51 AM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com wrote: On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com mailto:cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Fine by me (not that you need my approval !). How easy is it to build ATLAS targetting a specific CPU these days ? I think we need to at least support nosse and sse2 and above. I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I think Clint will have some time to help out next week. I did some analysis of SSE2 prevalence here: https://github.com/numpy/numpy/wiki/Window-versions Firefox crash reports now have about 1 percent of machines without SSE2. I suspect that people running new installs of numpy will have slightly better machines on average than Firefox users, but it's only a guess. I wonder if we could add a CPU check on numpy import to give a polite 'install from the exe' message for people without SSE2. We could, although you unfortunately can't do it easily from ctypes only (as you need some ASM). I can take a quick look at a simple cython extension that could be imported before anything else, and would raise an ImportError if the wrong arch is detected. assuming mingw is new enough #ifdef __SSE2___ raise_if(!__builtin_cpu_supports(sse)) #endof We need to support it for VS as well, but it looks like win32 API has a function to do it: http://msdn.microsoft.com/en-us/library/ms724482%28VS.85%29.aspx Makes it even easier. David in import_array() should do it ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
this is from: http://gcc.gnu.org/onlinedocs/gcc/X86-Built-in-Functions.html // ifunc resolvers fire before constructors, explicitly call the init function. __builtin_cpu_init (); if (__builtin_cpu_supports (ssse2)) code else code Cheers, Carl 2014-05-09 13:06 GMT+02:00 David Cournapeau courn...@gmail.com: On Fri, May 9, 2014 at 11:49 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 09.05.2014 12:42, David Cournapeau wrote: On Fri, May 9, 2014 at 1:51 AM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com wrote: On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com mailto:cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Fine by me (not that you need my approval !). How easy is it to build ATLAS targetting a specific CPU these days ? I think we need to at least support nosse and sse2 and above. I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I think Clint will have some time to help out next week. I did some analysis of SSE2 prevalence here: https://github.com/numpy/numpy/wiki/Window-versions Firefox crash reports now have about 1 percent of machines without SSE2. I suspect that people running new installs of numpy will have slightly better machines on average than Firefox users, but it's only a guess. I wonder if we could add a CPU check on numpy import to give a polite 'install from the exe' message for people without SSE2. We could, although you unfortunately can't do it easily from ctypes only (as you need some ASM). I can take a quick look at a simple cython extension that could be imported before anything else, and would raise an ImportError if the wrong arch is detected. assuming mingw is new enough #ifdef __SSE2___ raise_if(!__builtin_cpu_supports(sse)) #endof We need to support it for VS as well, but it looks like win32 API has a function to do it: http://msdn.microsoft.com/en-us/library/ms724482%28VS.85%29.aspx Makes it even easier. David in import_array() should do it ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau courn...@gmail.com wrote: On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.com wrote: Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Fine by me (not that you need my approval !). How easy is it to build ATLAS targetting a specific CPU these days ? I think we need to at least support nosse and sse2 and above. I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I think Clint will have some time to help out next week. I did some analysis of SSE2 prevalence here: https://github.com/numpy/numpy/wiki/Window-versions Firefox crash reports now have about 1 percent of machines without SSE2. I suspect that people running new installs of numpy will have slightly better machines on average than Firefox users, but it's only a guess. I wonder if we could add a CPU check on numpy import to give a polite 'install from the exe' message for people without SSE2. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Mon, Apr 28, 2014 at 10:54 AM, Chris Barker chris.bar...@noaa.gov wrote: On Sun, Apr 27, 2014 at 2:46 PM, Matthew Brett matthew.br...@gmail.com wrote: As you know, I'm really hoping it will be possible make a devkit for Python similar to the Ruby devkits [1]. That would be great! From a really quick glance, it looks like we could almost use the Ruby Devkit, maybe adding a couple add-ons.. Please, Carl correct me if I'm wrong, but I think the main issues are: 1) Linking to the right MSVC runtime for each Python version (seems to need different gcc specs) 2) Static linking - Carl's toolchain does full static linking including C runtimes so we don't need to deal with the DLL path. I'm not sure what the ruby devkit does to solve that. What do they do for 64 bit? It looks like they have their own Mingw-w64 toolchain. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
(1) Yes, Support for MSVC100 (python-3.3 and up) is on the TODO list (2) both toolchains are configured for static linking. No need to deploy: libgcc_s_dw2-1.dll, libgomp-1.dll, libquadmath-0.dll, libstdc++-6.dll, libgfortran-3.dll or libwinpthread-1.dll (3) I decided to create two dedicated toolchains for 32bit and for 64bit Regards, Carl 2014-04-29 11:19 GMT+02:00 Matthew Brett matthew.br...@gmail.com: Hi, On Mon, Apr 28, 2014 at 10:54 AM, Chris Barker chris.bar...@noaa.gov wrote: On Sun, Apr 27, 2014 at 2:46 PM, Matthew Brett matthew.br...@gmail.com wrote: As you know, I'm really hoping it will be possible make a devkit for Python similar to the Ruby devkits [1]. That would be great! From a really quick glance, it looks like we could almost use the Ruby Devkit, maybe adding a couple add-ons.. Please, Carl correct me if I'm wrong, but I think the main issues are: 1) Linking to the right MSVC runtime for each Python version (seems to need different gcc specs) 2) Static linking - Carl's toolchain does full static linking including C runtimes so we don't need to deal with the DLL path. I'm not sure what the ruby devkit does to solve that. What do they do for 64 bit? It looks like they have their own Mingw-w64 toolchain. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Matthew Brett matthew.br...@gmail.com wrote: 2) Static linking - Carl's toolchain does full static linking including C runtimes The C runtime cannot be statically linked. It would mean that we get multiple copies of errno and multiple malloc heaps in the process – one of each static CRT. We must use the same C runtime DLL as Python. But loading it is not a problem because Python has done that before NumPy is imported. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Correction: gcc (mingw) runtimes are statically linked. The C-runtime DLL msvcrXXX is linked dynamically. Carl 2014-04-29 17:10 GMT+02:00 Sturla Molden sturla.mol...@gmail.com: Matthew Brett matthew.br...@gmail.com wrote: 2) Static linking - Carl's toolchain does full static linking including C runtimes The C runtime cannot be statically linked. It would mean that we get multiple copies of errno and multiple malloc heaps in the process – one of each static CRT. We must use the same C runtime DLL as Python. But loading it is not a problem because Python has done that before NumPy is imported. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Mon, Apr 28, 2014 at 12:39 AM, Sturla Molden sturla.mol...@gmail.comwrote: Pauli Virtanen p...@iki.fi wrote: Yes, Windows is the only platform on which Fortran was problematic. OSX is somewhat saner in this respect. Oh yes, it seems there are official unofficial gfortran binaries available for OSX: http://gcc.gnu.org/wiki/GFortranBinaries#MacOS I'd be interested to hear if those work well for you. For people that just want to get things working, I would recommend to use the gfortran installers recommended at http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython. Those work for sure, and alternatives have usually proven to be problematic in the past. Ralf Cool :) Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Ralf Gommers ralf.gomm...@gmail.com wrote: I'd be interested to hear if those work well for you. For people that just want to get things working, I would recommend to use the gfortran installers recommended at a href=http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython.;http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython./a Those work for sure, and alternatives have usually proven to be problematic in the past. No problems thus far, but I only installed it yesterday. :-) I am not sure gcc-4.2 is needed anymore. Apple has retired it as platform C compiler on OS X. We need a Fortran compiler that can be used together with clang as C compiler. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Mon, Apr 28, 2014 at 6:06 PM, Sturla Molden sturla.mol...@gmail.comwrote: Ralf Gommers ralf.gomm...@gmail.com wrote: I'd be interested to hear if those work well for you. For people that just want to get things working, I would recommend to use the gfortran installers recommended at a href= http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython . http://scipy.org/scipylib/building/macosx.html#compilers-c-c-fortran-cython ./a Those work for sure, and alternatives have usually proven to be problematic in the past. No problems thus far, but I only installed it yesterday. :-) Sounds good. Let's give it a bit more time, once you've given it a good workout we can add that those gfortran 4.8.x compilers seem to work fine to the scipy build instructions. I am not sure gcc-4.2 is needed anymore. Apple has retired it as platform C compiler on OS X. We need a Fortran compiler that can be used together with clang as C compiler. Clang together with gfortran 4.2 works fine on OS X 10.9. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Ralf Gommers ralf.gomm...@gmail.com wrote: Sounds good. Let's give it a bit more time, once you've given it a good workout we can add that those gfortran 4.8.x compilers seem to work fine to the scipy build instructions. Yes, it needs to be tested properly. The build instructions for OS X Mavericks should also mention where to obtain Xcode (Appstore) and the secret command to retrieve the command-line utils after Xcode is installed: $ /usr/bin/xcode-select --install Probably it should also mention how to use alternative BLAS and LAPACK versions (MKL and OpenBLAS), although all three are equally performant on Mavericks (except Accelerate is not fork safe): https://twitter.com/nedlom/status/437427557919891457 Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Sun, Apr 27, 2014 at 2:46 PM, Matthew Brett matthew.br...@gmail.comwrote: As you know, I'm really hoping it will be possible make a devkit for Python similar to the Ruby devkits [1]. That would be great! From a really quick glance, it looks like we could almost use the Ruby Devkit, maybe adding a couple add-ons.. What do they do for 64 bit? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On 28/04/14 18:21, Ralf Gommers wrote:
No problems thus far, but I only installed it yesterday. :-)
Sounds good. Let's give it a bit more time, once you've given it a good
workout we can add that those gfortran 4.8.x compilers seem to work fine
to the scipy build instructions.
I have not looked at building SciPy yet, but I was able to build MPICH
3.0.4 from source without a problem. I worked on the first attempt
without any error or warnings. That is more than I hoped for...
Using BLAS and LAPACK from Accelerate also worked correctly with flags
-ff2c and -framework Accelerate. I can use it from Python (NumPy) with
ctypes and Cython. I get correct results and it does not segfault.
(It does segfault without -ff2c, but that is as expected, given that
Accelerate has f2c/g77 ABI.)
I was also able to build OpenBLAS with Clang as C compiler and gfortran
as Fortran compiler. It works correctly as well (both the build process
and the binaries I get).
So far it looks damn good :-)
The next step is to build NumPy and SciPy and run some tests :-)
Sturla
P.S. Here is what I did to build MPICH from source, for those interested:
$./configure CC=clang CXX=clang++ F77=gfortran FC=gfortran
--enable-fast=all,O3 --with-pm=gforker --prefix=/opt/mpich
$ make
$ sudo make install
$ export PATH=/opt/mpich/bin:$PATH # actually in ~/.bash_profile
Now testing with some hello worlds:
$ mpif77 -o hello hello.f
$ mpiexec -np 4 ./hello
Hello world
Hello world
Hello world
Hello world
$ rm hello
$ mpicc -o hello hello.c
$ mpiexec -np 4 ./hello
Hello world from process 0 of 4
Hello world from process 1 of 4
Hello world from process 2 of 4
Hello world from process 3 of 4
The hello world programs looked like this:
#include stdio.h
#include mpi.h
int main (int argc, char *argv[])
{
int rank, size;
MPI_Init (argc, argv);
MPI_Comm_rank (MPI_COMM_WORLD, rank);
MPI_Comm_size (MPI_COMM_WORLD, size);
printf( Hello world from process %d of %d\n, rank, size);
MPI_Finalize();
return 0;
}
program hello_world
include 'mpif.h'
integer ierr
call MPI_INIT(ierr)
print *, Hello world
call MPI_FINALIZE(ierr)
stop
end
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.comwrote: Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Fine by me (not that you need my approval !). How easy is it to build ATLAS targetting a specific CPU these days ? I think we need to at least support nosse and sse2 and above. David Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi Carl, On Sat, Apr 26, 2014 at 11:10 AM, Carl Kleffner cmkleff...@gmail.com wrote: Hi, basically the toolchain was created with a local fork of the mingw-builds build process along with some addons and patches. It is NOT a mingw-w64 fork. BTW: there are numerous mingw-w64 based toolchains out there, most of them build without any information about the build process and patches they used. As long as the mingw-builds maintainers continue working on their project, maintaining usuable toolchain for Python development on Windows should be feasible. More details are given here: http://article.gmane.org/gmane.comp.python.numeric.general/57446 I hope you don't mind, but I took the liberty of putting some of your email explanations and notes into the numpy wiki: https://github.com/numpy/numpy/wiki/Mingw-w64-faq https://github.com/numpy/numpy/wiki/Mingw-static-toolchain Do you have anywhere a description of what you did to create your fork of the build process? Maybe we can automate this using a Fedora or other cross-compiler? I think we need to make sure that the build system need not die if you get hired by some great company and can't work on this anymore. Do you think that is possible? Thanks again for the all the hard work you've done here. I think we are getting very close to a good solution, and that has seemed a long way off until now... Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, 25.04.2014 00:56, Matthew Brett kirjoitti: Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. Where can I get your numpy.patch scipy.patch and what's in them? Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Sun, Apr 27, 2014 at 6:09 AM, Pauli Virtanen p...@iki.fi wrote: Hi, 25.04.2014 00:56, Matthew Brett kirjoitti: Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. Where can I get your numpy.patch scipy.patch They are Carl's patches - here: https://bitbucket.org/carlkl/mingw-w64-for-python/downloads The scipy patch is tiny, the numpy patch more substantial. Carl - any interest into working up a pull-request for these? I'm happy to do it if you don't have time. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, I will definitly don't have not time until thursday this week working out the github workflow for a numpy pull request. So feel free to do it for me. BTW: There is a missing feature in the mingw-w64 toolchain. By now it features linking to msvcrt90 runtime only. I have do extend the specs file to allow linking to msvcr100 with an addional flag. Or create a dedicated toolchain - what do you think? Cheers, Carl 2014-04-27 20:29 GMT+02:00 Matthew Brett matthew.br...@gmail.com: Hi, On Sun, Apr 27, 2014 at 6:09 AM, Pauli Virtanen p...@iki.fi wrote: Hi, 25.04.2014 00:56, Matthew Brett kirjoitti: Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. Where can I get your numpy.patch scipy.patch They are Carl's patches - here: https://bitbucket.org/carlkl/mingw-w64-for-python/downloads The scipy patch is tiny, the numpy patch more substantial. Carl - any interest into working up a pull-request for these? I'm happy to do it if you don't have time. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Sun, Apr 27, 2014 at 2:34 PM, Carl Kleffner cmkleff...@gmail.com wrote: Hi, I will definitly don't have not time until thursday this week working out the github workflow for a numpy pull request. So feel free to do it for me. OK - I will have a go at this tomorrow. BTW: There is a missing feature in the mingw-w64 toolchain. By now it features linking to msvcrt90 runtime only. I have do extend the specs file to allow linking to msvcr100 with an addional flag. Or create a dedicated toolchain - what do you think? I don't know. Is this a discussion that should go to the mingw-w64 list do you think? It must be a very common feature. As you know, I'm really hoping it will be possible make a devkit for Python similar to the Ruby devkits [1]. The ideal would be a devkit that transparently picked up 32 vs 64 bit, and MSVC runtime according to the Python version. For example, OSX compilation automatically picks up the OSX SDK with which the relevant Python was built. Do you think something like this is possible? That would be a great improvement for people building extensions and wheels on Windows. Cheers, Matthew [1] http://rubyinstaller.org/add-ons/devkit/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I can also give access to the dedicated machine doing the builds. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Sun, Apr 27, 2014 at 6:06 PM, Carl Kleffner cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. What I did in the past is just to download the ATLAS binaries from the scipy/numpy wiki and move them into the Python/Dlls directory. IIRC My impression was that finding ATLAS binaries was the difficult part, not moving them into the right directory. Cheers, Carl 2014-04-27 23:46 GMT+02:00 Matthew Brett matthew.br...@gmail.com: Hi, On Sun, Apr 27, 2014 at 2:34 PM, Carl Kleffner cmkleff...@gmail.com wrote: Hi, I will definitly don't have not time until thursday this week working out the github workflow for a numpy pull request. So feel free to do it for me. OK - I will have a go at this tomorrow. BTW: There is a missing feature in the mingw-w64 toolchain. By now it features linking to msvcrt90 runtime only. I have do extend the specs file to allow linking to msvcr100 with an addional flag. Or create a dedicated toolchain - what do you think? I don't know. Is this a discussion that should go to the mingw-w64 list do you think? It must be a very common feature. As you know, I'm really hoping it will be possible make a devkit for Python similar to the Ruby devkits [1]. I got my entire initial setup on the computer I'm using right now through python-xy, including MingW 32. The only thing I ever had to do was to create the `distutils.cfg` in new python install. python-xy relies on the availability of a open source development environment for numpy and scipy, and has been restricted so far to python 32 versions. winpython is only a python distribution and is also available for 64bit (with Gohlke binaries I think) I think it would be very helpful to get python-xy set up for development for 64 bit versions, now that the toolchain with MingW is available. I'm skeptical about having lot's of distributions that install all their own full toolchain (I always worry about which one is actually on the path. I deleted my first git for Windows version because it came with a built-in MSYS/MingW toolchain and now just use the nice and small portable version.) The ideal would be a devkit that transparently picked up 32 vs 64 bit, and MSVC runtime according to the Python version. For example, OSX compilation automatically picks up the OSX SDK with which the relevant Python was built. Do you think something like this is possible? That would be a great improvement for people building extensions and wheels on Windows. How does MingW64 decide whether to build 32 or to build 64 bit versions? Does the python version matter for MingW? or should this pick up one of the Visual SDK's that the user needs to install? Josef Cheers, Matthew [1] http://rubyinstaller.org/add-ons/devkit/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Pauli Virtanen p...@iki.fi wrote: Yes, Windows is the only platform on which Fortran was problematic. OSX is somewhat saner in this respect. Oh yes, it seems there are official unofficial gfortran binaries available for OSX: http://gcc.gnu.org/wiki/GFortranBinaries#MacOS Cool :) Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Aha, On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com wrote: A possible option is to install the toolchain inside site-packages and to deploy it as PYPI wheel or wininst packages. The PATH to the toolchain could be extended during import of the package. But I have no idea, whats the best strategy to additionaly install ATLAS or other third party libraries. Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the devkit package. It sounds like OpenBLAS will be much easier to build, so we could start with ATLAS binaries as a default, expecting OpenBLAS to be built more often with the toolchain. I think that's how numpy binary installers are built at the moment - using old binary builds of ATLAS. I'm happy to provide the builds of ATLAS - e.g. here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds I just found the official numpy binary builds of ATLAS: https://github.com/numpy/vendor/tree/master/binaries But - they are from an old version of ATLAS / Lapack, and only for 32-bit. David - what say we update these to latest ATLAS stable? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, On Thu, Apr 24, 2014 at 5:26 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:29 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Apr 24, 2014 at 5:08 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 Strange. That second one looks familiar, at least the -96 part. Wonder why this doesn't show up with the MKL builds. ok tried again, this time deleting the old numpy directories before installing Ran 4760 tests in 42.124s OK (KNOWNFAIL=10, SKIP=8) nose.result.TextTestResult run=4760 errors=0 failures=0 so pip also seems to be reusing leftover files. all clear. Running the statsmodels test suite, I get a failure in test_discrete.TestProbitCG where fmin_cg converges to something that differs in the 3rd decimal. I usually only test the 32-bit version, so I don't know if this is specific to this scipy version, but we haven't seen this in a long time. I used our nightly binaries http://statsmodels.sourceforge.net/binaries/ That's interesting, you saw also we're getting failures on the tests for powell optimization because of small unit-at-last-place differences in the exp function in mingw-w64. Is there any chance you can track down where the optimization path is diverging and why? It's just that - if this is also the exp function maybe we can see if the error is exceeding reasonable bounds and then feed back to mingw-w64 and fall back to the numpy default implementation in the
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Fri, Apr 25, 2014 at 1:21 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Apr 24, 2014 at 5:26 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:29 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Apr 24, 2014 at 5:08 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 Strange. That second one looks familiar, at least the -96 part. Wonder why this doesn't show up with the MKL builds. ok tried again, this time deleting the old numpy directories before installing Ran 4760 tests in 42.124s OK (KNOWNFAIL=10, SKIP=8) nose.result.TextTestResult run=4760 errors=0 failures=0 so pip also seems to be reusing leftover files. all clear. Running the statsmodels test suite, I get a failure in test_discrete.TestProbitCG where fmin_cg converges to something that differs in the 3rd decimal. I usually only test the 32-bit version, so I don't know if this is specific to this scipy version, but we haven't seen this in a long time. I used our nightly binaries http://statsmodels.sourceforge.net/binaries/ That's interesting, you saw also we're getting failures on the tests for powell optimization because of small unit-at-last-place differences in the exp function in mingw-w64. Is there any chance you can track down
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Sat, Apr 26, 2014 at 10:10 AM, josef.p...@gmail.com wrote: On Fri, Apr 25, 2014 at 1:21 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Apr 24, 2014 at 5:26 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:29 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Apr 24, 2014 at 5:08 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installersnumpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 Strange. That second one looks familiar, at least the -96 part. Wonder why this doesn't show up with the MKL builds. ok tried again, this time deleting the old numpy directories before installing Ran 4760 tests in 42.124s OK (KNOWNFAIL=10, SKIP=8) nose.result.TextTestResult run=4760 errors=0 failures=0 so pip also seems to be reusing leftover files. all clear. Running the statsmodels test suite, I get a failure in test_discrete.TestProbitCG where fmin_cg converges to something that differs in the 3rd decimal. I usually only test the 32-bit version, so I don't know if this is specific to this scipy version, but we haven't seen this in a long time. I used our nightly binaries http://statsmodels.sourceforge.net/binaries/ That's interesting, you saw also we're getting failures on the tests for powell optimization because of small unit-at-last-place differences in
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Sat, Apr 26, 2014 at 10:20 AM, josef.p...@gmail.com wrote: On Sat, Apr 26, 2014 at 10:10 AM, josef.p...@gmail.com wrote: On Fri, Apr 25, 2014 at 1:21 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Apr 24, 2014 at 5:26 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:29 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Apr 24, 2014 at 5:08 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installersnumpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 Strange. That second one looks familiar, at least the -96 part. Wonder why this doesn't show up with the MKL builds. ok tried again, this time deleting the old numpy directories before installing Ran 4760 tests in 42.124s OK (KNOWNFAIL=10, SKIP=8) nose.result.TextTestResult run=4760 errors=0 failures=0 so pip also seems to be reusing leftover files. all clear. Running the statsmodels test suite, I get a failure in test_discrete.TestProbitCG where fmin_cg converges to something that differs in the 3rd decimal. I usually only test the 32-bit version, so I don't know if this is specific to this scipy version, but we haven't seen this in a long time. I used our nightly binaries http://statsmodels.sourceforge.net/binaries/ That's interesting, you saw also we're getting failures on the tests for
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Matthew Brett matthew.br...@gmail.com wrote: Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. Thanks for your great effort to solve this mess. By Murphy's law, I do not have access to a Windows computer on which to test now. :-( This approach worries me a bit though: Will we have to maintain a fork of MinGW-w64 for building NumPy and SciPy? Should this toolset be distributed along with NumPy and SciPy on Windows? I presume it is needed to build C and Cython extensions? On the positive side: Does this mean we finally can use gfortran on Windows? And if so, can we use Fortran versions beyond Fortran 77 in SciPy now? Or is Mac OS X a blocker? Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
25.04.2014 08:57, Sturla Molden kirjoitti: [clip] On the positive side: Does this mean we finally can use gfortran on Windows? And if so, can we use Fortran versions beyond Fortran 77 in SciPy now? Or is Mac OS X a blocker? Yes, Windows is the only platform on which Fortran was problematic. OSX is somewhat saner in this respect. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi, basically the toolchain was created with a local fork of the mingw-builds build process along with some addons and patches. It is NOT a mingw-w64 fork. BTW: there are numerous mingw-w64 based toolchains out there, most of them build without any information about the build process and patches they used. As long as the mingw-builds maintainers continue working on their project, maintaining usuable toolchain for Python development on Windows should be feasible. More details are given here: http://article.gmane.org/gmane.comp.python.numeric.general/57446 Regards Carl 2014-04-25 7:57 GMT+02:00 Sturla Molden sturla.mol...@gmail.com: Matthew Brett matthew.br...@gmail.com wrote: Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. Thanks for your great effort to solve this mess. By Murphy's law, I do not have access to a Windows computer on which to test now. :-( This approach worries me a bit though: Will we have to maintain a fork of MinGW-w64 for building NumPy and SciPy? Should this toolset be distributed along with NumPy and SciPy on Windows? I presume it is needed to build C and Cython extensions? On the positive side: Does this mean we finally can use gfortran on Windows? And if so, can we use Fortran versions beyond Fortran 77 in SciPy now? Or is Mac OS X a blocker? Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Fri, Apr 25, 2014 at 12:00 AM, Charles R Harris charlesr.har...@gmail.com wrote: Cool. After all these long years... Now all we need is a box running tests for CI. There is http://www.appveyor.com/ though I haven't tried doing anything with it yet... (yes it says .NET at the top, but then at the bottom it says that this is a lie and it doesn't care what kind of project you have) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Thu, Apr 24, 2014 at 7:08 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy scipy looks good, just two powell trace failures Josef Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Thu, Apr 24, 2014 at 5:08 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 Strange. That second one looks familiar, at least the -96 part. Wonder why this doesn't show up with the MKL builds. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Apr 24, 2014 at 5:08 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 Strange. That second one looks familiar, at least the -96 part. Wonder why this doesn't show up with the MKL builds. ok tried again, this time deleting the old numpy directories before installing Ran 4760 tests in 42.124s OK (KNOWNFAIL=10, SKIP=8) nose.result.TextTestResult run=4760 errors=0 failures=0 so pip also seems to be reusing leftover files. all clear. Josef Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
OT: Oh, I hate pip and packages that require numpy E:\tmpC:\Python27\python C:\Python27\Scripts\pip-script.py install -U patsy Downloading/unpacking patsy Running setup.py (path:c:\users\josef\appdata\local\temp\pip_build_josef\patsy\setup.py) egg_info for package patsy no previously-included directories found matching 'doc\_build' Downloading/unpacking numpy from https://pypi.python.org/packages/source/n/numpy/numpy-1.8.1.tar.gz#md5=be95babe263bfa3428363d6db5b64678(from patsy) Found existing installation: numpy 1.6.2 Uninstalling numpy: Successfully uninstalled numpy Running setup.py install for numpy ... ... C:\Python27\lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'define_macros' warnings.warn(msg) error: Unable to find vcvarsall.bat Rolling back uninstall of numpy Cleaning up... - user error ? I have a stale numpy-1.6.2-py2.7.egg-info file in site-packages put that's a nice new feature of pip Rolling back uninstall of numpy numpy is still here Josef ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Thu, Apr 24, 2014 at 7:29 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:20 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Apr 24, 2014 at 5:08 PM, josef.p...@gmail.com wrote: On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I get two test failures with numpy Josef np.test() Running unit tests for numpy NumPy version 1.8.1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] nose version 1.1.2 == FAIL: test_iterator.test_iter_broadcasting_errors -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 657, in test_iter_broadcasting_errors '(2)-(2,newaxis)') % msg) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError: Message operands could not be broadcast together with remapped shapes [original-remapped]: (2,3)-(2,3) (2,)-(2,newaxis) and requested shape (4,3) doesn't contain remapped operand shape(2)-(2,newaxis) == FAIL: test_iterator.test_iter_array_cast -- Traceback (most recent call last): File C:\Python27\lib\site-packages\nose\case.py, line 197, in runTest self.test(*self.arg) File C:\Python27\lib\site-packages\numpy\core\tests\test_iterator.py, line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 255, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose) File C:\Python27\lib\site-packages\numpy\testing\utils.py, line 317, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96L DESIRED: -96 -- Ran 4828 tests in 46.306s FAILED (KNOWNFAIL=10, SKIP=8, failures=2) nose.result.TextTestResult run=4828 errors=0 failures=2 Strange. That second one looks familiar, at least the -96 part. Wonder why this doesn't show up with the MKL builds. ok tried again, this time deleting the old numpy directories before installing Ran 4760 tests in 42.124s OK (KNOWNFAIL=10, SKIP=8) nose.result.TextTestResult run=4760 errors=0 failures=0 so pip also seems to be reusing leftover files. all clear. Running the statsmodels test suite, I get a failure in test_discrete.TestProbitCG where fmin_cg converges to something that differs in the 3rd decimal. I usually only test the 32-bit version, so I don't know if this is specific to this scipy version, but we haven't seen this in a long time. I used our nightly binaries http://statsmodels.sourceforge.net/binaries/ Josef Josef Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing
On Thu, Apr 24, 2014 at 7:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi Matthew, On Thu, Apr 24, 2014 at 3:56 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, Thanks to Cark Kleffner's toolchain and some help from Clint Whaley (main author of ATLAS), I've built 64-bit windows numpy and scipy wheels for testing. The build uses Carl's custom mingw-w64 build with static linking. There are two harmless test failures on scipy (being discussed on the list at the moment) - tests otherwise clean. Wheels are here: https://nipy.bic.berkeley.edu/scipy_installers/numpy-1.8.1-cp27-none-win_amd64.whl https://nipy.bic.berkeley.edu/scipy_installers/scipy-0.13.3-cp27-none-win_amd64.whl You can test with: pip install -U pip # to upgrade pip to latest pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy scipy Please do send feedback. ATLAS binary here: https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds/atlas-64-full-sse2.tar.bz2 Many thanks for Carl in particular for doing all the hard work, Cool. After all these long years... Now all we need is a box running tests for CI. Very good news, after 3 years interruption I might be able to build scipy again, and switch now to a 64bit development version. Thanks for pushing for this, and doing all the hard work. Josef Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
