Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
Julian Taylor jtaylor.debian at googlemail.com writes: [clip] There haven't been any real complaints from applications yet, only testsuite failure of scipy. Either the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet. What is broken is calculating eigenvalues of complex-valued sparse matrices and iterative solution of complex-valued linear equations. I.e., nothing obscure. A likely explanation is that win32 + Numpy 1.9 is a less common platform, and users whose code started failing just infrequently do not report bugs as easily... The majority of f2py should still be working, numpys own f2py testsuite passes on win32. Perhaps the arrays are aligned by chance? I don't think the test suite repeats the complex valued intent(inout) parameter test many times. [clip] I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created. David's test case is this: n = 4 x = np.zeros(n * 3, dtype=D) _dummy.zfoo(x, n) where the argument is declared as double complex, dimension(3*n), intent(inout) in f2py. The ARPACK stuff in Scipy also does pretty much just this. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 11/26/2014 12:50 AM, Andrea Gavana wrote: On 25 November 2014 at 19:33, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com wrote: On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com wrote: David Cournapeau courn...@gmail.com mailto:courn...@gmail.com wrote: Shall we consider a href=https://github.com/scipy/scipy/issues/4168; https://github.com/scipy/scipy/issues/4168/a to be a blocker (the issue arises on scipy master as well as 0.14.1) ? It is really bad, but does anyone know what is really going on? Yes, it is in the bug report. Nice move. I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? yes, before the change numpy accepted a factor 10 performance regression in complex indexing to satisfy sparc. Dozens of f2py compiled extensions are going to fail soon here - which I'll reluctantly check tomorrow. I don't want to think about other people on Win32 facing the same issue elsewhere... :-) There haven't been any real complaints from applications yet, only testsuite failure of scipy. Either the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet. The majority of f2py should still be working, numpys own f2py testsuite passes on win32. I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created. It should of course be fixed, I have an old branch with aligned allocators lying around somewhere which should fix the issue or at least give users a way to work around it. The lesson I take from this is that we need a test ;) Also, that it would be nice if we got more testing by users before releases. But things are as they are, this will get fixed, and we will move on with one more lesson learned. If anyone is to blame, it is the reviewer, namely, myself. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
On 11/26/2014 02:19 PM, Charles R Harris wrote: On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com wrote: On 11/26/2014 12:50 AM, Andrea Gavana wrote: On 25 November 2014 at 19:33, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com mailto:courn...@gmail.com mailto:courn...@gmail.com wrote: On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com wrote: David Cournapeau courn...@gmail.com mailto:courn...@gmail.com mailto:courn...@gmail.com mailto:courn...@gmail.com wrote: Shall we consider a href=https://github.com/scipy/scipy/issues/4168;https://github.com/scipy/scipy/issues/4168/a to be a blocker (the issue arises on scipy master as well as 0.14.1) ? It is really bad, but does anyone know what is really going on? Yes, it is in the bug report. Nice move. I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? yes, before the change numpy accepted a factor 10 performance regression in complex indexing to satisfy sparc. Dozens of f2py compiled extensions are going to fail soon here - which I'll reluctantly check tomorrow. I don't want to think about other people on Win32 facing the same issue elsewhere... :-) There haven't been any real complaints from applications yet, only testsuite failure of scipy. Eithe the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet. The majority of f2py should still be working, numpys own f2py testsuite passes on win32. I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created. It should of course be fixed, I have an old branch with aligned allocators lying around somewhere which should fix the issue or at least give users a way to work around it. The lesson I take from this is that we need a test ;) Also, that it would be nice if we got more testing by users before releases. But things are as they are, this will get fixed, and we will move on with one more lesson learned. If anyone is to blame, it is the reviewer, namely, myself. concerning actually fixing it I guess we have 3 options: 1. reduce maximum copy alignment from currently 16 to 8 on platforms that need it. That will automatically reduce the needed alignment of complex on win32 to 8 bytes. Do other compilers provide something similar to gccs __BIGGEST_ALIGNMENT__? 2. remove bloating of complex alignment and let sparc crash. 3. add an aligned allocator I somewhat favor 1, it has the risk that a vectorizing compiler might wreak havoc but to my knowledge numpy does not actually have real 16 byte copies. There are some occurrences of npy_int128, but those likely are not used on windows. fwiw I could reproduce the issue on linux i386 with -malign-double, (I could have sworn I tested that configuration too...) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
On Wed, Nov 26, 2014 at 6:59 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 11/26/2014 02:19 PM, Charles R Harris wrote: On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com wrote: On 11/26/2014 12:50 AM, Andrea Gavana wrote: On 25 November 2014 at 19:33, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com mailto:courn...@gmail.com mailto:courn...@gmail.com wrote: On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com wrote: David Cournapeau courn...@gmail.com mailto: courn...@gmail.com mailto:courn...@gmail.com mailto:courn...@gmail.com wrote: Shall we consider a href=https://github.com/scipy/scipy/issues/4168; https://github.com/scipy/scipy/issues/4168/a to be a blocker (the issue arises on scipy master as well as 0.14.1) ? It is really bad, but does anyone know what is really going on? Yes, it is in the bug report. Nice move. I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? yes, before the change numpy accepted a factor 10 performance regression in complex indexing to satisfy sparc. Dozens of f2py compiled extensions are going to fail soon here - which I'll reluctantly check tomorrow. I don't want to think about other people on Win32 facing the same issue elsewhere... :-) There haven't been any real complaints from applications yet, only testsuite failure of scipy. Eithe the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet. The majority of f2py should still be working, numpys own f2py testsuite passes on win32. I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created. It should of course be fixed, I have an old branch with aligned allocators lying around somewhere which should fix the issue or at least give users a way to work around it. The lesson I take from this is that we need a test ;) Also, that it would be nice if we got more testing by users before releases. But things are as they are, this will get fixed, and we will move on with one more lesson learned. If anyone is to blame, it is the reviewer, namely, myself. concerning actually fixing it I guess we have 3 options: 1. reduce maximum copy alignment from currently 16 to 8 on platforms that need it. That will automatically reduce the needed alignment of complex on win32 to 8 bytes. Do other compilers provide something similar to gccs __BIGGEST_ALIGNMENT__? 2. remove bloating of complex alignment and let sparc crash. 3. add an aligned allocator I somewhat favor 1, it has the risk that a vectorizing compiler might wreak havoc but to my knowledge numpy does not actually have real 16 byte copies. There are some occurrences of npy_int128, but those likely are not used on windows. fwiw I could reproduce the issue on linux i386 with -malign-double, (I could have sworn I tested that configuration too...) I would also go for 1) on the general principal that alignment needs to be platform dependent. I suppose the problem here is that is might also be compiler dependent. Where do the aligned copies take place? What is the downside to that solution? Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
On 26 Nov 2014 13:59, Julian Taylor jtaylor.deb...@googlemail.com wrote: concerning actually fixing it I guess we have 3 options: 1. reduce maximum copy alignment from currently 16 to 8 on platforms that need it. That will automatically reduce the needed alignment of complex on win32 to 8 bytes. Do other compilers provide something similar to gccs __BIGGEST_ALIGNMENT__? 2. remove bloating of complex alignment and let sparc crash. 3. add an aligned allocator I somewhat favor 1, it has the risk that a vectorizing compiler might wreak havoc but to my knowledge numpy does not actually have real 16 byte copies. There are some occurrences of npy_int128, but those likely are not used on windows. fwiw I could reproduce the issue on linux i386 with -malign-double, (I could have sworn I tested that configuration too...) I'm not sure what maximum copy alignment means in this context, but (1) does sound the most like a proper fix to me too. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
David Cournapeau courn...@gmail.com wrote: Shall we consider a href=https://github.com/scipy/scipy/issues/4168;https://github.com/scipy/scipy/issues/4168/a to be a blocker (the issue arises on scipy master as well as 0.14.1) ? It is really bad, but does anyone know what is really going on? Which changes to NumPy set this off? Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden sturla.mol...@gmail.com wrote: David Cournapeau courn...@gmail.com wrote: Shall we consider a href=https://github.com/scipy/scipy/issues/4168; https://github.com/scipy/scipy/issues/4168/a to be a blocker (the issue arises on scipy master as well as 0.14.1) ? It is really bad, but does anyone know what is really going on? Yes, it is in the bug report. David Which changes to NumPy set this off? Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
On 25 November 2014 at 19:33, David Cournapeau courn...@gmail.com wrote: On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden sturla.mol...@gmail.com wrote: David Cournapeau courn...@gmail.com wrote: Shall we consider a href=https://github.com/scipy/scipy/issues/4168; https://github.com/scipy/scipy/issues/4168/a to be a blocker (the issue arises on scipy master as well as 0.14.1) ? It is really bad, but does anyone know what is really going on? Yes, it is in the bug report. Nice move. I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? Dozens of f2py compiled extensions are going to fail soon here - which I'll reluctantly check tomorrow. I don't want to think about other people on Win32 facing the same issue elsewhere... :-) Happy hacking. Andrea. Imagination Is The Only Weapon In The War Against Reality. http://www.infinity77.net # - # def ask_mailing_list_support(email): if mention_platform_and_version() and include_sample_app(): send_message(email) else: install_malware() erase_hard_drives() # - # ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release
On Tue, Nov 25, 2014 at 11:50 PM, Andrea Gavana andrea.gav...@gmail.com wrote: On 25 November 2014 at 19:33, David Cournapeau courn...@gmail.com wrote: On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden sturla.mol...@gmail.com wrote: David Cournapeau courn...@gmail.com wrote: Shall we consider a href=https://github.com/scipy/scipy/issues/4168;https://github.com/scipy/scipy/issues/4168/a to be a blocker (the issue arises on scipy master as well as 0.14.1) ? It is really bad, but does anyone know what is really going on? Yes, it is in the bug report. Nice move. I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? It's a regression in 1.9, yes; 1.8 is fine. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
