Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On Fri, Mar 22, 2013 at 10:39 PM, Colin J. Williams cjwilliam...@gmail.comwrote: On 20/03/2013 11:12 AM, Frédéric Bastien wrote: On Wed, Mar 20, 2013 at 11:01 AM, Colin J. Williamscjwilliam...@gmail.com cjwilliam...@gmail.com wrote: On 20/03/2013 10:30 AM, Frédéric Bastien wrote: Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? 32 bits. That explain why you have memory problem but not other people with 64 bits version. So if you want to work with bigger input, change to a python 64 bits. Fred Thanks to the people who responded to my report that numpy, with Python 3.2 was significantly slower than with Python 2.7. I have updated to numpy 1.7.0 for each of the Pythons 2.7.3, 3.2.3 and 3.3.0. The Pythons came from python.org and the Numpys from PyPi. The SciPy site still points to Source Forge, I gathered from the responses that Source Forge is no longer recommended for downloads. That's not the case. The official binaries for NumPy and SciPy are on SourceForge. The Windows installers on PyPI are there to make easy_install work, but they're likely slower than the SF installers (no SSE2/SSE3 instructions). Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On Sat, Mar 23, 2013 at 12:21 PM, Ralf Gommers ralf.gomm...@gmail.comwrote: That's not the case. The official binaries for NumPy and SciPy are on SourceForge. The Windows installers on PyPI are there to make easy_install work, but they're likely slower than the SF installers (no SSE2/SSE3 instructions). Ralf Is there a reason why the same binaries can't be used for both? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On Sat, Mar 23, 2013 at 12:23 PM, Todd toddr...@gmail.com wrote: On Sat, Mar 23, 2013 at 12:21 PM, Ralf Gommers ralf.gomm...@gmail.comwrote: That's not the case. The official binaries for NumPy and SciPy are on SourceForge. The Windows installers on PyPI are there to make easy_install work, but they're likely slower than the SF installers (no SSE2/SSE3 instructions). Ralf Is there a reason why the same binaries can't be used for both? The SF .exe superpack installers contains three installers: plain, SSE2 and SSE3 support. easy_install doesn't know what to do with such an installer. See http://thread.gmane.org/gmane.comp.python.numeric.general/29395/focus=29582for the discussion on why things are as they are now. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On 23/03/2013 7:21 AM, Ralf Gommers wrote: On Fri, Mar 22, 2013 at 10:39 PM, Colin J. Williams cjwilliam...@gmail.com wrote: On 20/03/2013 11:12 AM, Frédéric Bastien wrote: On Wed, Mar 20, 2013 at 11:01 AM, Colin J. Williams cjwilliam...@gmail.com wrote: On 20/03/2013 10:30 AM, Frédéric Bastien wrote: Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? 32 bits. That explain why you have memory problem but not other people with 64 bits version. So if you want to work with bigger input, change to a python 64 bits. Fred Thanks to the people who responded to my report that numpy, with Python 3.2 was significantly slower than with Python 2.7. I have updated to numpy 1.7.0 for each of the Pythons 2.7.3, 3.2.3 and 3.3.0. The Pythons came from python.org and the Numpys from PyPi. The SciPy site still points to Source Forge, I gathered from the responses that Source Forge is no longer recommended for downloads. That's not the case. The official binaries for NumPy and SciPy are on SourceForge. The Windows installers on PyPI are there to make easy_install work, but they're likely slower than the SF installers (no SSE2/SSE3 instructions). Ralf Thanks, I'll read over Robert Kern's comments. PyPi is the simpler process, but, if the result is unoptimized code, then easy_install is not the way to go. The code is available here(http://web.ncf.ca/cjw/testFPSpeed.py) and the most recent test results are here(http://web.ncf.ca/cjw/FP%2023-Mar-13%20Test%20Summary.txt). These are using PyPi, I'll look into SourceForge. Colin W. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
I am a bit worried about the differences in results. Just to be sure you are comparing apples with apples, it may be a good idea to set the seed at the beginning: np.random.seed( SEED ) where SEED is an int. This way, you will be inverting always the same matrix, regardless of the Python version. I think, even if the timing is different, the results should be the same. http://docs.scipy.org/doc/numpy/reference/generated/numpy.random.seed.html#numpy.random.seed David. On 23 March 2013 15:39, Colin J. Williams cjwilliam...@gmail.com wrote: On 23/03/2013 7:21 AM, Ralf Gommers wrote: On Fri, Mar 22, 2013 at 10:39 PM, Colin J. Williams cjwilliam...@gmail.com wrote: On 20/03/2013 11:12 AM, Frédéric Bastien wrote: On Wed, Mar 20, 2013 at 11:01 AM, Colin J. Williams cjwilliam...@gmail.com wrote: On 20/03/2013 10:30 AM, Frédéric Bastien wrote: Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? 32 bits. That explain why you have memory problem but not other people with 64 bits version. So if you want to work with bigger input, change to a python 64 bits. Fred Thanks to the people who responded to my report that numpy, with Python 3.2 was significantly slower than with Python 2.7. I have updated to numpy 1.7.0 for each of the Pythons 2.7.3, 3.2.3 and 3.3.0. The Pythons came from python.org and the Numpys from PyPi. The SciPy site still points to Source Forge, I gathered from the responses that Source Forge is no longer recommended for downloads. That's not the case. The official binaries for NumPy and SciPy are on SourceForge. The Windows installers on PyPI are there to make easy_install work, but they're likely slower than the SF installers (no SSE2/SSE3 instructions). Ralf Thanks, I'll read over Robert Kern's comments. PyPi is the simpler process, but, if the result is unoptimized code, then easy_install is not the way to go. The code is available here(http://web.ncf.ca/cjw/testFPSpeed.py) and the most recent test results are here(http://web.ncf.ca/cjw/FP%2023-Mar-13%20Test%20Summary.txt). These are using PyPi, I'll look into SourceForge. Colin W. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On 23/03/2013 12:05 AM, Chris Barker - NOAA Federal wrote: On Fri, Mar 22, 2013 at 2:39 PM, Colin J. Williams cjwilliam...@gmail.com wrote: I have updated to numpy 1.7.0 for each of the Pythons 2.7.3, 3.2.3 and 3.3.0. ... The tests, which are available here(http://web.ncf.ca/cjw/FP%20Summary%20over%20273-323-330.txt), show that 3.2 is slower, but not to the same degree reported before. Have posted your test code anywhere? Anyway, depending on how you did your timings, that looks to me like 3.* is a bit faster with small data, and pretty much within measurement error for the large datasets. And if the large ones are doing things with really big arrays (I'm assuming pretty big, as you're getting close to 32 bit memory limits...), then it's really hard to imagine how python version could make a noticeable difference -- the real work would be in the numpy code, and that's exactly the same on all python versions. If you are using BLAS or LAPACK stuff, then there might be some differences with the different builds, though I wouldn't expect so if you ar getting them from the same source. -Chris I used the versions from PyPi, this choice has been questioned. I'll compare with the SourceForge versions. Also, I shall be incorporating the random SEED. I expect to report the results in the next week or so. The test code used is available here: http://web.ncf.ca/cjw/testFPSpeed.py Colin W. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On 20/03/2013 11:12 AM, Frédéric Bastien wrote: On Wed, Mar 20, 2013 at 11:01 AM, Colin J. Williams cjwilliam...@gmail.com wrote: On 20/03/2013 10:30 AM, Frédéric Bastien wrote: Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? 32 bits. That explain why you have memory problem but not other people with 64 bits version. So if you want to work with bigger input, change to a python 64 bits. Fred Thanks to the people who responded to my report that numpy, with Python 3.2 was significantly slower than with Python 2.7. I have updated to numpy 1.7.0 for each of the Pythons 2.7.3, 3.2.3 and 3.3.0. The Pythons came from python.org and the Numpys from PyPi. The SciPy site still points to Source Forge, I gathered from the responses that Source Forge is no longer recommended for downloads. The tests, which are available here(http://web.ncf.ca/cjw/FP%20Summary%20over%20273-323-330.txt), show that 3.2 is slower, but not to the same degree reported before. Colin W. PS There seems also to be a Python problem with the treatment of sys.argv in Python 3.3 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On Fri, Mar 22, 2013 at 2:39 PM, Colin J. Williams cjwilliam...@gmail.com wrote: I have updated to numpy 1.7.0 for each of the Pythons 2.7.3, 3.2.3 and 3.3.0. ... The tests, which are available here(http://web.ncf.ca/cjw/FP%20Summary%20over%20273-323-330.txt), show that 3.2 is slower, but not to the same degree reported before. Have posted your test code anywhere? Anyway, depending on how you did your timings, that looks to me like 3.* is a bit faster with small data, and pretty much within measurement error for the large datasets. And if the large ones are doing things with really big arrays (I'm assuming pretty big, as you're getting close to 32 bit memory limits...), then it's really hard to imagine how python version could make a noticeable difference -- the real work would be in the numpy code, and that's exactly the same on all python versions. If you are using BLAS or LAPACK stuff, then there might be some differences with the different builds, though I wouldn't expect so if you ar getting them from the same source. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
Without much detailed knowledge of the topic, I would expect both versions to give very similar timing, as it is essentially a call to ATLAS function, not much is done in Python. Given this, maybe the difference is in ATLAS itself. How have you installed it? When you compile ATLAS, it will do some machine-specific optimisation, but if you have installed a binary chances are that your version is optimised for a machine quite different from yours. So, two different installations could have been compiled in different machines and so one is more suited for your machine. If you want to be sure, I would try to compile ATLAS (this may be difficult) or check the same on a very different machine (like an AMD processor, different architecture...). Just for reference, on Linux Python 2.7 64 bits can deal with these matrices easily. %timeit mat=np.random.random((6143,6143)); matinv= np.linalg.inv(mat); res = np.dot(mat, matinv); diff= res-np.eye(6143); print np.sum(np.abs(diff)) 2.41799631031e-05 1.13955868701e-05 3.64338191541e-05 1.13484781021e-05 1 loops, best of 3: 156 s per loop Intel i5, 4 GB of RAM and SSD. ATLAS installed from Fedora repository (I don't run heavy stuff on this computer). On 20 March 2013 14:46, Colin J. Williams c...@ncf.ca wrote: I have a small program which builds random matrices for increasing matrix orders, inverts the matrix and checks the precision of the product. At some point, one would expect operations to fail, when the memory capacity is exceeded. In both Python 2.7 and 3.2 matrices of order 3,071 area handled, but not 6,143. Using wall-clock times, with win32, Python 3.2 is slower than Python 2.7. The profiler indicates a problem in the solver. Done on a Pentium, with 2.7 GHz processor, 2 GB of RAM and 221 GB of free disk space. Both Python 3.2.3 and Python 2.7.3 use numpy 1.6.2. The results are show below. Colin W. _ 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.097 Time elapsed (seconds)= 0.004143 order=5 measure ofimprecision= 2.207 Time elapsed (seconds)= 0.001514 order= 11 measure ofimprecision= 2.372 Time elapsed (seconds)= 0.001455 order= 23 measure ofimprecision= 3.318 Time elapsed (seconds)= 0.001608 order= 47 measure ofimprecision= 4.257 Time elapsed (seconds)= 0.002339 order= 95 measure ofimprecision= 4.986 Time elapsed (seconds)= 0.005747 order= 191 measure ofimprecision= 5.788 Time elapsed (seconds)= 0.029974 order= 383 measure ofimprecision= 6.765 Time elapsed (seconds)= 0.145339 order= 767 measure ofimprecision= 7.909 Time elapsed (seconds)= 0.841142 order= 1535 measure ofimprecision= 8.532 Time elapsed (seconds)= 5.793630 order= 3071 measure ofimprecision= 9.774 Time elapsed (seconds)= 39.559540 order= 6143 Process terminated by a MemoryError Above: 2.7.3 Below: Python 3.2.3 bbb_bbb 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.000 Time elapsed (seconds)= 0.113930 order=5 measure ofimprecision= 1.807 Time elapsed (seconds)= 0.001373 order= 11 measure ofimprecision= 2.395 Time elapsed (seconds)= 0.001468 order= 23 measure ofimprecision= 3.073 Time elapsed (seconds)= 0.001609 order= 47 measure ofimprecision= 5.642 Time elapsed (seconds)= 0.002687 order= 95 measure ofimprecision= 5.745 Time elapsed (seconds)= 0.013510 order= 191 measure ofimprecision= 5.866 Time elapsed (seconds)= 0.061560 order= 383 measure ofimprecision= 7.129 Time elapsed (seconds)= 0.418490 order= 767 measure ofimprecision= 8.240 Time elapsed (seconds)= 3.815713 order= 1535 measure ofimprecision= 8.735 Time elapsed (seconds)= 27.877270 order= 3071 measure ofimprecision= 9.996 Time elapsed (seconds)=212.545610 order= 6143 Process terminated by a MemoryError ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
Hi, Could also be that they are linked to different libs such as atlas and standart Blas. What is the output of numpy.show_config() in the two different python versions. Jens On Wed, Mar 20, 2013 at 2:14 PM, Daπid davidmen...@gmail.com wrote: Without much detailed knowledge of the topic, I would expect both versions to give very similar timing, as it is essentially a call to ATLAS function, not much is done in Python. Given this, maybe the difference is in ATLAS itself. How have you installed it? When you compile ATLAS, it will do some machine-specific optimisation, but if you have installed a binary chances are that your version is optimised for a machine quite different from yours. So, two different installations could have been compiled in different machines and so one is more suited for your machine. If you want to be sure, I would try to compile ATLAS (this may be difficult) or check the same on a very different machine (like an AMD processor, different architecture...). Just for reference, on Linux Python 2.7 64 bits can deal with these matrices easily. %timeit mat=np.random.random((6143,6143)); matinv= np.linalg.inv(mat); res = np.dot(mat, matinv); diff= res-np.eye(6143); print np.sum(np.abs(diff)) 2.41799631031e-05 1.13955868701e-05 3.64338191541e-05 1.13484781021e-05 1 loops, best of 3: 156 s per loop Intel i5, 4 GB of RAM and SSD. ATLAS installed from Fedora repository (I don't run heavy stuff on this computer). On 20 March 2013 14:46, Colin J. Williams c...@ncf.ca wrote: I have a small program which builds random matrices for increasing matrix orders, inverts the matrix and checks the precision of the product. At some point, one would expect operations to fail, when the memory capacity is exceeded. In both Python 2.7 and 3.2 matrices of order 3,071 area handled, but not 6,143. Using wall-clock times, with win32, Python 3.2 is slower than Python 2.7. The profiler indicates a problem in the solver. Done on a Pentium, with 2.7 GHz processor, 2 GB of RAM and 221 GB of free disk space. Both Python 3.2.3 and Python 2.7.3 use numpy 1.6.2. The results are show below. Colin W. _ 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.097 Time elapsed (seconds)= 0.004143 order=5 measure ofimprecision= 2.207 Time elapsed (seconds)= 0.001514 order= 11 measure ofimprecision= 2.372 Time elapsed (seconds)= 0.001455 order= 23 measure ofimprecision= 3.318 Time elapsed (seconds)= 0.001608 order= 47 measure ofimprecision= 4.257 Time elapsed (seconds)= 0.002339 order= 95 measure ofimprecision= 4.986 Time elapsed (seconds)= 0.005747 order= 191 measure ofimprecision= 5.788 Time elapsed (seconds)= 0.029974 order= 383 measure ofimprecision= 6.765 Time elapsed (seconds)= 0.145339 order= 767 measure ofimprecision= 7.909 Time elapsed (seconds)= 0.841142 order= 1535 measure ofimprecision= 8.532 Time elapsed (seconds)= 5.793630 order= 3071 measure ofimprecision= 9.774 Time elapsed (seconds)= 39.559540 order= 6143 Process terminated by a MemoryError Above: 2.7.3 Below: Python 3.2.3 bbb_bbb 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.000 Time elapsed (seconds)= 0.113930 order=5 measure ofimprecision= 1.807 Time elapsed (seconds)= 0.001373 order= 11 measure ofimprecision= 2.395 Time elapsed (seconds)= 0.001468 order= 23 measure ofimprecision= 3.073 Time elapsed (seconds)= 0.001609 order= 47 measure ofimprecision= 5.642 Time elapsed (seconds)= 0.002687 order= 95 measure ofimprecision= 5.745 Time elapsed (seconds)= 0.013510 order= 191 measure ofimprecision= 5.866 Time elapsed (seconds)= 0.061560 order= 383 measure ofimprecision= 7.129 Time elapsed (seconds)= 0.418490 order= 767 measure ofimprecision= 8.240 Time elapsed (seconds)= 3.815713 order= 1535 measure ofimprecision= 8.735 Time elapsed (seconds)= 27.877270 order= 3071 measure ofimprecision= 9.996 Time elapsed (seconds)=212.545610 order= 6143 Process terminated by a MemoryError ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? Fred On Wed, Mar 20, 2013 at 10:14 AM, Daπid davidmen...@gmail.com wrote: Without much detailed knowledge of the topic, I would expect both versions to give very similar timing, as it is essentially a call to ATLAS function, not much is done in Python. Given this, maybe the difference is in ATLAS itself. How have you installed it? When you compile ATLAS, it will do some machine-specific optimisation, but if you have installed a binary chances are that your version is optimised for a machine quite different from yours. So, two different installations could have been compiled in different machines and so one is more suited for your machine. If you want to be sure, I would try to compile ATLAS (this may be difficult) or check the same on a very different machine (like an AMD processor, different architecture...). Just for reference, on Linux Python 2.7 64 bits can deal with these matrices easily. %timeit mat=np.random.random((6143,6143)); matinv= np.linalg.inv(mat); res = np.dot(mat, matinv); diff= res-np.eye(6143); print np.sum(np.abs(diff)) 2.41799631031e-05 1.13955868701e-05 3.64338191541e-05 1.13484781021e-05 1 loops, best of 3: 156 s per loop Intel i5, 4 GB of RAM and SSD. ATLAS installed from Fedora repository (I don't run heavy stuff on this computer). On 20 March 2013 14:46, Colin J. Williams c...@ncf.ca wrote: I have a small program which builds random matrices for increasing matrix orders, inverts the matrix and checks the precision of the product. At some point, one would expect operations to fail, when the memory capacity is exceeded. In both Python 2.7 and 3.2 matrices of order 3,071 area handled, but not 6,143. Using wall-clock times, with win32, Python 3.2 is slower than Python 2.7. The profiler indicates a problem in the solver. Done on a Pentium, with 2.7 GHz processor, 2 GB of RAM and 221 GB of free disk space. Both Python 3.2.3 and Python 2.7.3 use numpy 1.6.2. The results are show below. Colin W. _ 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.097 Time elapsed (seconds)= 0.004143 order=5 measure ofimprecision= 2.207 Time elapsed (seconds)= 0.001514 order= 11 measure ofimprecision= 2.372 Time elapsed (seconds)= 0.001455 order= 23 measure ofimprecision= 3.318 Time elapsed (seconds)= 0.001608 order= 47 measure ofimprecision= 4.257 Time elapsed (seconds)= 0.002339 order= 95 measure ofimprecision= 4.986 Time elapsed (seconds)= 0.005747 order= 191 measure ofimprecision= 5.788 Time elapsed (seconds)= 0.029974 order= 383 measure ofimprecision= 6.765 Time elapsed (seconds)= 0.145339 order= 767 measure ofimprecision= 7.909 Time elapsed (seconds)= 0.841142 order= 1535 measure ofimprecision= 8.532 Time elapsed (seconds)= 5.793630 order= 3071 measure ofimprecision= 9.774 Time elapsed (seconds)= 39.559540 order= 6143 Process terminated by a MemoryError Above: 2.7.3 Below: Python 3.2.3 bbb_bbb 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.000 Time elapsed (seconds)= 0.113930 order=5 measure ofimprecision= 1.807 Time elapsed (seconds)= 0.001373 order= 11 measure ofimprecision= 2.395 Time elapsed (seconds)= 0.001468 order= 23 measure ofimprecision= 3.073 Time elapsed (seconds)= 0.001609 order= 47 measure ofimprecision= 5.642 Time elapsed (seconds)= 0.002687 order= 95 measure ofimprecision= 5.745 Time elapsed (seconds)= 0.013510 order= 191 measure ofimprecision= 5.866 Time elapsed (seconds)= 0.061560 order= 383 measure ofimprecision= 7.129 Time elapsed (seconds)= 0.418490 order= 767 measure ofimprecision= 8.240 Time elapsed (seconds)= 3.815713 order= 1535 measure ofimprecision= 8.735 Time elapsed (seconds)= 27.877270 order= 3071 measure ofimprecision= 9.996 Time elapsed (seconds)=212.545610 order= 6143 Process terminated by a MemoryError ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On 20/03/2013 10:14 AM, Daπid wrote: Without much detailed knowledge of the topic, I would expect both versions to give very similar timing, as it is essentially a call to ATLAS function, not much is done in Python. Given this, maybe the difference is in ATLAS itself. How have you installed it? I know nothing about what goes on behind the scenes. I am using the win32 binary package. Colin W. When you compile ATLAS, it will do some machine-specific optimisation, but if you have installed a binary chances are that your version is optimised for a machine quite different from yours. So, two different installations could have been compiled in different machines and so one is more suited for your machine. If you want to be sure, I would try to compile ATLAS (this may be difficult) or check the same on a very different machine (like an AMD processor, different architecture...). Just for reference, on Linux Python 2.7 64 bits can deal with these matrices easily. %timeit mat=np.random.random((6143,6143)); matinv= np.linalg.inv(mat); res = np.dot(mat, matinv); diff= res-np.eye(6143); print np.sum(np.abs(diff)) 2.41799631031e-05 1.13955868701e-05 3.64338191541e-05 1.13484781021e-05 1 loops, best of 3: 156 s per loop Intel i5, 4 GB of RAM and SSD. ATLAS installed from Fedora repository (I don't run heavy stuff on this computer). On 20 March 2013 14:46, Colin J. Williams c...@ncf.ca wrote: I have a small program which builds random matrices for increasing matrix orders, inverts the matrix and checks the precision of the product. At some point, one would expect operations to fail, when the memory capacity is exceeded. In both Python 2.7 and 3.2 matrices of order 3,071 area handled, but not 6,143. Using wall-clock times, with win32, Python 3.2 is slower than Python 2.7. The profiler indicates a problem in the solver. Done on a Pentium, with 2.7 GHz processor, 2 GB of RAM and 221 GB of free disk space. Both Python 3.2.3 and Python 2.7.3 use numpy 1.6.2. The results are show below. Colin W. _ 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.097 Time elapsed (seconds)= 0.004143 order=5 measure ofimprecision= 2.207 Time elapsed (seconds)= 0.001514 order= 11 measure ofimprecision= 2.372 Time elapsed (seconds)= 0.001455 order= 23 measure ofimprecision= 3.318 Time elapsed (seconds)= 0.001608 order= 47 measure ofimprecision= 4.257 Time elapsed (seconds)= 0.002339 order= 95 measure ofimprecision= 4.986 Time elapsed (seconds)= 0.005747 order= 191 measure ofimprecision= 5.788 Time elapsed (seconds)= 0.029974 order= 383 measure ofimprecision= 6.765 Time elapsed (seconds)= 0.145339 order= 767 measure ofimprecision= 7.909 Time elapsed (seconds)= 0.841142 order= 1535 measure ofimprecision= 8.532 Time elapsed (seconds)= 5.793630 order= 3071 measure ofimprecision= 9.774 Time elapsed (seconds)= 39.559540 order= 6143 Process terminated by a MemoryError Above: 2.7.3 Below: Python 3.2.3 bbb_bbb 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.000 Time elapsed (seconds)= 0.113930 order=5 measure ofimprecision= 1.807 Time elapsed (seconds)= 0.001373 order= 11 measure ofimprecision= 2.395 Time elapsed (seconds)= 0.001468 order= 23 measure ofimprecision= 3.073 Time elapsed (seconds)= 0.001609 order= 47 measure ofimprecision= 5.642 Time elapsed (seconds)= 0.002687 order= 95 measure ofimprecision= 5.745 Time elapsed (seconds)= 0.013510 order= 191 measure ofimprecision= 5.866 Time elapsed (seconds)= 0.061560 order= 383 measure ofimprecision= 7.129 Time elapsed (seconds)= 0.418490 order= 767 measure ofimprecision= 8.240 Time elapsed (seconds)= 3.815713 order= 1535 measure ofimprecision= 8.735 Time elapsed (seconds)= 27.877270 order= 3071 measure ofimprecision= 9.996 Time elapsed (seconds)=212.545610 order= 6143 Process terminated by a MemoryError ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On 20/03/2013 10:29 AM, Jens Nielsen
wrote:
Hi,
Could also be that they are linked to different
libs such as atlas and standart Blas. What is the output of
numpy.show_config() in the two different python
versions.
Jens
Thanks for this pointer.
The result for Py2.7:
numpy.show_config()
atlas_threads_info:
NOT AVAILABLE
blas_opt_info:
libraries = ['f77blas', 'cblas', 'atlas']
library_dirs = ['C:\\local\\lib\\yop\\sse3']
define_macros = [('NO_ATLAS_INFO', -1)]
language = c
atlas_blas_threads_info:
NOT AVAILABLE
lapack_opt_info:
libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
library_dirs = ['C:\\local\\lib\\yop\\sse3']
define_macros = [('NO_ATLAS_INFO', -1)]
language = f77
atlas_info:
libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
library_dirs = ['C:\\local\\lib\\yop\\sse3']
define_macros = [('NO_ATLAS_INFO', -1)]
language = f77
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
atlas_blas_info:
libraries = ['f77blas', 'cblas', 'atlas']
library_dirs = ['C:\\local\\lib\\yop\\sse3']
define_macros = [('NO_ATLAS_INFO', -1)]
language = c
mkl_info:
NOT AVAILABLE
The result for 3.2:
import numpy
numpy.show_config()
lapack_info:
NOT AVAILABLE
lapack_opt_info:
NOT AVAILABLE
blas_info:
NOT AVAILABLE
atlas_threads_info:
NOT AVAILABLE
blas_src_info:
NOT AVAILABLE
atlas_blas_info:
NOT AVAILABLE
lapack_src_info:
NOT AVAILABLE
atlas_blas_threads_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
blas_opt_info:
NOT AVAILABLE
atlas_info:
NOT AVAILABLE
lapack_mkl_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE
I hope that this helps.
Colin W.
On Wed, Mar 20, 2013 at 2:14 PM, Daπid
davidmen...@gmail.com
wrote:
Without
much detailed knowledge of the topic, I would expect both
versions to give very similar timing, as it is essentially a
call to
ATLAS function, not much is done in Python.
Given this, maybe the difference is in ATLAS itself. How
have you
installed it? When you compile ATLAS, it will do some
machine-specific
optimisation, but if you have installed a binary chances are
that your
version is optimised for a machine quite different from
yours. So, two
different installations could have been compiled in
different machines
and so one is more suited for your machine. If you want to
be sure, I
would try to compile ATLAS (this may be difficult) or check
the same
on a very different machine (like an AMD processor,
different
architecture...).
Just for reference, on Linux Python 2.7 64 bits can deal
with these
matrices easily.
%timeit mat=np.random.random((6143,6143)); matinv=
np.linalg.inv(mat);
res = np.dot(mat, matinv); diff= res-np.eye(6143); print
np.sum(np.abs(diff))
2.41799631031e-05
1.13955868701e-05
3.64338191541e-05
1.13484781021e-05
1 loops, best of 3: 156 s per loop
Intel i5, 4 GB of RAM and SSD. ATLAS installed from Fedora
repository
(I don't run heavy stuff on this computer).
On 20 March 2013 14:46, Colin J. Williams c...@ncf.ca
wrote:
I have a small program which builds random matrices
for increasing matrix
orders, inverts the matrix and checks the precision
of the product. At some
point, one would expect operations to fail, when
the memory capacity is
exceeded. In both Python 2.7 and 3.2 matrices of
order 3,071 area handled,
but not 6,143.
Using wall-clock times, with win32, Python 3.2 is
slower
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On 20/03/2013 10:30 AM, Frédéric Bastien wrote: Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? 32 bits. Colin W. Fred On Wed, Mar 20, 2013 at 10:14 AM, Daπid davidmen...@gmail.com wrote: Without much detailed knowledge of the topic, I would expect both versions to give very similar timing, as it is essentially a call to ATLAS function, not much is done in Python. Given this, maybe the difference is in ATLAS itself. How have you installed it? When you compile ATLAS, it will do some machine-specific optimisation, but if you have installed a binary chances are that your version is optimised for a machine quite different from yours. So, two different installations could have been compiled in different machines and so one is more suited for your machine. If you want to be sure, I would try to compile ATLAS (this may be difficult) or check the same on a very different machine (like an AMD processor, different architecture...). Just for reference, on Linux Python 2.7 64 bits can deal with these matrices easily. %timeit mat=np.random.random((6143,6143)); matinv= np.linalg.inv(mat); res = np.dot(mat, matinv); diff= res-np.eye(6143); print np.sum(np.abs(diff)) 2.41799631031e-05 1.13955868701e-05 3.64338191541e-05 1.13484781021e-05 1 loops, best of 3: 156 s per loop Intel i5, 4 GB of RAM and SSD. ATLAS installed from Fedora repository (I don't run heavy stuff on this computer). On 20 March 2013 14:46, Colin J. Williams c...@ncf.ca wrote: I have a small program which builds random matrices for increasing matrix orders, inverts the matrix and checks the precision of the product. At some point, one would expect operations to fail, when the memory capacity is exceeded. In both Python 2.7 and 3.2 matrices of order 3,071 area handled, but not 6,143. Using wall-clock times, with win32, Python 3.2 is slower than Python 2.7. The profiler indicates a problem in the solver. Done on a Pentium, with 2.7 GHz processor, 2 GB of RAM and 221 GB of free disk space. Both Python 3.2.3 and Python 2.7.3 use numpy 1.6.2. The results are show below. Colin W. _ 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.097 Time elapsed (seconds)= 0.004143 order=5 measure ofimprecision= 2.207 Time elapsed (seconds)= 0.001514 order= 11 measure ofimprecision= 2.372 Time elapsed (seconds)= 0.001455 order= 23 measure ofimprecision= 3.318 Time elapsed (seconds)= 0.001608 order= 47 measure ofimprecision= 4.257 Time elapsed (seconds)= 0.002339 order= 95 measure ofimprecision= 4.986 Time elapsed (seconds)= 0.005747 order= 191 measure ofimprecision= 5.788 Time elapsed (seconds)= 0.029974 order= 383 measure ofimprecision= 6.765 Time elapsed (seconds)= 0.145339 order= 767 measure ofimprecision= 7.909 Time elapsed (seconds)= 0.841142 order= 1535 measure ofimprecision= 8.532 Time elapsed (seconds)= 5.793630 order= 3071 measure ofimprecision= 9.774 Time elapsed (seconds)= 39.559540 order= 6143 Process terminated by a MemoryError Above: 2.7.3 Below: Python 3.2.3 bbb_bbb 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.000 Time elapsed (seconds)= 0.113930 order=5 measure ofimprecision= 1.807 Time elapsed (seconds)= 0.001373 order= 11 measure ofimprecision= 2.395 Time elapsed (seconds)= 0.001468 order= 23 measure ofimprecision= 3.073 Time elapsed (seconds)= 0.001609 order= 47 measure ofimprecision= 5.642 Time elapsed (seconds)= 0.002687 order= 95 measure ofimprecision= 5.745 Time elapsed (seconds)= 0.013510 order= 191 measure ofimprecision= 5.866 Time elapsed (seconds)= 0.061560 order= 383 measure ofimprecision= 7.129 Time elapsed (seconds)= 0.418490 order= 767 measure ofimprecision= 8.240 Time elapsed (seconds)= 3.815713 order= 1535 measure ofimprecision= 8.735 Time elapsed (seconds)= 27.877270 order= 3071 measure ofimprecision= 9.996 Time elapsed (seconds)=212.545610 order= 6143 Process terminated by a MemoryError ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
The python3 version is compiled without any optimised library and is falling back on a slow version. Where did you get this installation from? Jens On Wed, Mar 20, 2013 at 3:01 PM, Colin J. Williams cjwilliam...@gmail.comwrote: On 20/03/2013 10:30 AM, Frédéric Bastien wrote: Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? 32 bits. Colin W. Fred On Wed, Mar 20, 2013 at 10:14 AM, Daπid davidmen...@gmail.com wrote: Without much detailed knowledge of the topic, I would expect both versions to give very similar timing, as it is essentially a call to ATLAS function, not much is done in Python. Given this, maybe the difference is in ATLAS itself. How have you installed it? When you compile ATLAS, it will do some machine-specific optimisation, but if you have installed a binary chances are that your version is optimised for a machine quite different from yours. So, two different installations could have been compiled in different machines and so one is more suited for your machine. If you want to be sure, I would try to compile ATLAS (this may be difficult) or check the same on a very different machine (like an AMD processor, different architecture...). Just for reference, on Linux Python 2.7 64 bits can deal with these matrices easily. %timeit mat=np.random.random((6143,6143)); matinv= np.linalg.inv(mat); res = np.dot(mat, matinv); diff= res-np.eye(6143); print np.sum(np.abs(diff)) 2.41799631031e-05 1.13955868701e-05 3.64338191541e-05 1.13484781021e-05 1 loops, best of 3: 156 s per loop Intel i5, 4 GB of RAM and SSD. ATLAS installed from Fedora repository (I don't run heavy stuff on this computer). On 20 March 2013 14:46, Colin J. Williams c...@ncf.ca wrote: I have a small program which builds random matrices for increasing matrix orders, inverts the matrix and checks the precision of the product. At some point, one would expect operations to fail, when the memory capacity is exceeded. In both Python 2.7 and 3.2 matrices of order 3,071 area handled, but not 6,143. Using wall-clock times, with win32, Python 3.2 is slower than Python 2.7. The profiler indicates a problem in the solver. Done on a Pentium, with 2.7 GHz processor, 2 GB of RAM and 221 GB of free disk space. Both Python 3.2.3 and Python 2.7.3 use numpy 1.6.2. The results are show below. Colin W. _ 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.097 Time elapsed (seconds)= 0.004143 order=5 measure ofimprecision= 2.207 Time elapsed (seconds)= 0.001514 order= 11 measure ofimprecision= 2.372 Time elapsed (seconds)= 0.001455 order= 23 measure ofimprecision= 3.318 Time elapsed (seconds)= 0.001608 order= 47 measure ofimprecision= 4.257 Time elapsed (seconds)= 0.002339 order= 95 measure ofimprecision= 4.986 Time elapsed (seconds)= 0.005747 order= 191 measure ofimprecision= 5.788 Time elapsed (seconds)= 0.029974 order= 383 measure ofimprecision= 6.765 Time elapsed (seconds)= 0.145339 order= 767 measure ofimprecision= 7.909 Time elapsed (seconds)= 0.841142 order= 1535 measure ofimprecision= 8.532 Time elapsed (seconds)= 5.793630 order= 3071 measure ofimprecision= 9.774 Time elapsed (seconds)= 39.559540 order= 6143 Process terminated by a MemoryError Above: 2.7.3 Below: Python 3.2.3 bbb_bbb 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] order=2 measure ofimprecision= 0.000 Time elapsed (seconds)= 0.113930 order=5 measure ofimprecision= 1.807 Time elapsed (seconds)= 0.001373 order= 11 measure ofimprecision= 2.395 Time elapsed (seconds)= 0.001468 order= 23 measure ofimprecision= 3.073 Time elapsed (seconds)= 0.001609 order= 47 measure ofimprecision= 5.642 Time elapsed (seconds)= 0.002687 order= 95 measure ofimprecision= 5.745 Time elapsed (seconds)= 0.013510 order= 191 measure ofimprecision= 5.866 Time elapsed (seconds)= 0.061560 order= 383 measure ofimprecision= 7.129 Time elapsed (seconds)= 0.418490 order= 767 measure ofimprecision= 8.240 Time elapsed (seconds)= 3.815713 order= 1535 measure ofimprecision= 8.735 Time elapsed (seconds)= 27.877270 order= 3071 measure ofimprecision= 9.996 Time elapsed (seconds)=212.545610 order= 6143 Process terminated by a MemoryError ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On Wed, Mar 20, 2013 at 11:01 AM, Colin J. Williams cjwilliam...@gmail.com wrote: On 20/03/2013 10:30 AM, Frédéric Bastien wrote: Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? 32 bits. That explain why you have memory problem but not other people with 64 bits version. So if you want to work with bigger input, change to a python 64 bits. Fred ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On 20/03/2013 11:06 AM, Jens Nielsen wrote: The python3 version is compiled without any optimised library and is falling back on a slow version. Where did you get this installation from? Jens From the SciPy site. Colin W. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On 20/03/2013 11:12 AM, Frédéric Bastien wrote: On Wed, Mar 20, 2013 at 11:01 AM, Colin J. Williams cjwilliam...@gmail.com wrote: On 20/03/2013 10:30 AM, Frédéric Bastien wrote: Hi, win32 do not mean it is a 32 bits windows. sys.platform always return win32 on 32bits and 64 bits windows even for python 64 bits. But that is a good question, is your python 32 or 64 bits? 32 bits. That explain why you have memory problem but not other people with 64 bits version. So if you want to work with bigger input, change to a python 64 bits. Fred But my machine is only 32 bit. Colin W. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
