Re: [Numpy-discussion] import numpy is slow
On Jul 31, 2008, at 3:53 AM, David Cournapeau wrote: You are supposed to run the tests on an installed numpy, not in the sources: import numpy numpy.test(verbose = 10) Doesn't that make things more cumbersome to test? That is, if I were to make a change I would need to: - python setup.py build (to put the code into the build/* subdirectory) - cd the build directory, or switch to a terminal which was already there - manually do the import/test code you wrote, or a write two-line program for it I would rather do 'nosetests' in the source tree, if at all feasible, although that might only be possible for the Python source. Hmm. And it looks like testing/nosetester.py (which implements the 'test' function above) is meant to make it easier to run nose, except my feeling is the extra level of wrapping makes things more complicated. The nosetest command-line appears to be more flexible, with support for, for examples, dropping into the debugger on errors, and reseting the coverage test files. I'm speaking out of ignorance, btw. Cheers, Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Example of numpy cov() not correct?
Wed, 30 Jul 2008 18:49:10 +0300, Nadav Horesh wrote: If you read the cov function documentation you'll see that if a second vector is given, it joins the 2 into one matrix and calculate the covariance of it. In your case, you are looking for the off-diagonal elements. So the final answer to the OP's question is: Yes, the example on http://www.scipy.org/Numpy_Example_List_With_Doc#cov is wrong; cov(T,P) indeed returns a matrix. And it would be nice if someone fixed this, you can simply register a wiki account and fix the problem. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
Hi All, I've been reading this discussion with interest. I would just to highlight an alternate use of numpy to interactive use. We have a cluster of machines which process tasks on an individual basis where a master tasks may spawn 600 slave tasks to be processed. These tasks are spread across the cluster and processed as scripts in a individual python thread. Although reducing the process time by 300 seconds for the master task is only about a 1.5% speedup (total time can be i excess of 24000s). We process large number of these tasks in any given year and every little helps! Hanni 2008/7/31 Stéfan van der Walt [EMAIL PROTECTED] 2008/7/31 Andrew Dalke [EMAIL PROTECTED]: The user base for numpy might be .. 10,000 people? 100,000 people? Let's go with the latter, and assume that with command-line scripts, CGI scripts, and the other programs that people write in order to help do research means that numpy is started on average 10 times a day. 100,000 people * 10 times / day * 0.1 seconds per startup = almost 28 people-hours spent each day waiting for numpy to start. I don't buy that argument. No single person is agile enough to do anything useful in the half a second or so it takes to start up NumPy. No one is *waiting* for NumPy to start. Just by answering this e-mail I could have (and maybe should have) started NumPy three hundred and sixty times. I don't want to argue about this, though. Write the patches, file a ticket, and hopefully someone will deem them important enough to apply them. Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 7:31 AM, Hanni Ali [EMAIL PROTECTED] wrote: I would just to highlight an alternate use of numpy to interactive use. We have a cluster of machines which process tasks on an individual basis where a master tasks may spawn 600 slave tasks to be processed. These tasks are spread across the cluster and processed as scripts in a individual python thread. Although reducing the process time by 300 seconds for the master task is only about a 1.5% speedup (total time can be i excess of 24000s). We process large number of these tasks in any given year and every little helps! There are other components of NumPy/SciPy that are more worthy of optimization. Given that programmer time is a scarce resource, it's more sensible to direct our efforts towards making the other 98.5% of the computation faster. /law of diminishing returns -- Nathan Bell [EMAIL PROTECTED] http://graphics.cs.uiuc.edu/~wnbell/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
Nathan Bell wrote: There are other components of NumPy/SciPy that are more worthy of optimization. Given that programmer time is a scarce resource, it's more sensible to direct our efforts towards making the other 98.5% of the computation faster. To be fair, when I took a look at the problem last month, it took a few of us (Robert and me IIRC) maximum 2 man hours altogether to divide by two numpy import times on linux, without altering at all the API. Maybe there are more things which can be done to get to a more 'flat' profile. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 5:36 AM, Andrew Dalke [EMAIL PROTECTED] wrote: The user base for numpy might be .. 10,000 people? 100,000 people? Let's go with the latter, and assume that with command-line scripts, CGI scripts, and the other programs that people write in order to help do research means that numpy is started on average 10 times a day. 100,000 people * 10 times / day * 0.1 seconds per startup = almost 28 people-hours spent each day waiting for numpy to start. I'm willing to spend a few days to achieve that. Perhaps there's fewer people than I'm estimating. OTOH, perhaps there are more imports of numpy per day. An order of magnitude less time is still a couple of hours each day as the world waits to import all of the numpy libraries. If on average people import numpy 10 times a day and it could be made 0.1 seconds faster then that's 1 second per person per day. If it takes on average 5 minutes to learn to import the module directly and the onus is all on numpy, then after 1 year of use the efficiency has made up for it, and the benefits continue to grow. Just think of the savings that could be achieved if all 2.1 million Walmart employees were outfitted with colostomy bags. 0.5 hours / day for bathroom breaks * 2,100,000 employees * 365 days/year * $7/hour = $2,682,750,000/year Granted, I'm probably not the first to run these numbers. -- Nathan Bell [EMAIL PROTECTED] http://graphics.cs.uiuc.edu/~wnbell/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] The date/time dtype and the casting issue
A Thursday 31 July 2008, Matt Knox escrigué: While on the topic of FAME... being a financial analyst, I really am quite fond of the multitude of quarterly frequencies we have in the timeseries package (with different year end points) because they are very useful when doing things like calenderizing earnings from companies with different fiscal year ends. On Thu, 31 Jul 2008, Francesc Alted apparently wrote: Well, introducing a quarter should not be difficult. We just wanted to keep the set of supported time units under a minimum (the list is already quite large). We thought that the quarter fits better as a 'derived' time unit, similarly as biweekly, semester or biyearly (to name just a few examples). However, if quarters are found to be much more important than other derived time units, they can go into the proposal too. Quarterly frequency is probably the most analyzed frequency in macroeconometrics. Widely used macroeconometrics packages (e.g., EViews) traditionally support only three explicit frequencies: annual, quarterly, and monthly. Cheers, Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 03:41:15PM +0900, David Cournapeau wrote: Yes. Nothing that an easy make file cannot solve, nonetheless (I am sure I am not the only one with a makefile/script which automates the above, to test a new svn updated numpy in one command). That's why distutils have a test target. You can do python setup.py test, and if you have setup you setup.py properly it should work (obviously it is easy to make this statement, and harder to get the thing working). Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 12:43:17PM +0200, Andrew Dalke wrote: Startup performance has not been a numpy concern. It a concern for me, and it has been (for other packages) a concern for some of my clients. I am curious, if startup performance is a problem, I guess it is because you are running lots of little scripts where startup time is big compared to run time. Did you think of forking them from an already started process. I had this same problem (with libraries way slower than numpy to load) and used os.fork to a great success. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Example of numpy cov() not correct?
On Thu, Jul 31, 2008 at 1:14 AM, Pauli Virtanen [EMAIL PROTECTED] wrote: Yes, the example on http://www.scipy.org/Numpy_Example_List_With_Doc#cov is wrong; cov(T,P) indeed returns a matrix. And it would be nice if someone fixed this, you can simply register a wiki account and fix the problem. Done. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 2:12 AM, Andrew Dalke [EMAIL PROTECTED] wrote: Hmm. And it looks like testing/nosetester.py (which implements the 'test' function above) is meant to make it easier to run nose, except my feeling is the extra level of wrapping makes things more complicated. The nosetest command-line appears to be more flexible, with support for, for examples, dropping into the debugger on errors, and reseting the coverage test files. You can actually pass those sorts of options to nose through the extra_argv parameter in test(). That might be a little cumbersome, but (as far as I know) it's something I'm going to do so infrequently it's not a big deal. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
Gael Varoquaux wrote: That's why distutils have a test target. You can do python setup.py test, and if you have setup you setup.py properly it should work (obviously it is easy to make this statement, and harder to get the thing working). I have already seen some discussion about distutils like this, if you mean something like this: http://blog.ianbicking.org/pythons-makefile.html but I would take with rake and make over this anytime. I just don't understand why something like rake does not exist in python, but well, let's not go there. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 11:05:33PM +0900, David Cournapeau wrote: Gael Varoquaux wrote: That's why distutils have a test target. You can do python setup.py test, and if you have setup you setup.py properly it should work (obviously it is easy to make this statement, and harder to get the thing working). I have already seen some discussion about distutils like this, if you mean something like this: http://blog.ianbicking.org/pythons-makefile.html but I would take with rake and make over this anytime. I just don't understand why something like rake does not exist in python, but well, let's not go there. Well, actually, in the enthought tools suite we use setuptools for packaging (I don't want to start a controversy, I am not advocating the use of setuptools, just stating a fact) and nose for testing, and getting setup.py test to wrok, including do the build test and download nose if not there, is a matter of addig those two lines to the setup.py: tests_require = [ 'nose = 0.10.3', ], test_suite = 'nose.collector', Obviously, the build part has to be well-tuned for the machinery to work, but there is a lot of value here. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
Gael Varoquaux wrote: Obviously, the build part has to be well-tuned for the machinery to work, but there is a lot of value here. Ah yes, setuptools does have this. But this is specific to setuptools, bare distutils does not have this test command, right ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 10:14 AM, Gael Varoquaux [EMAIL PROTECTED] wrote: On Thu, Jul 31, 2008 at 12:43:17PM +0200, Andrew Dalke wrote: Startup performance has not been a numpy concern. It a concern for me, and it has been (for other packages) a concern for some of my clients. I am curious, if startup performance is a problem, I guess it is because you are running lots of little scripts where startup time is big compared to run time. Did you think of forking them from an already started process. I had this same problem (with libraries way slower than numpy to load) and used os.fork to a great success. Start up time is an issue for me, but in a larger sense than just numpy. I do run many scripts, some that are ephemeral and some that take significant amounts of time. However, numpy is just one of many many libraries that I must import, so improvements, even minor ones, are appreciated. The morale of this discussion, for me, is that just because _you_ don't care about a particular aspect or feature, doesn't mean that others don't or shouldn't. Your workarounds may not be viable for me and vice-versa. So let's just go with the spirit of open source and encourage those motivated to controbute to do so, provided their suggestions are sensible and do not break code. -Kevin ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 11:16:12PM +0900, David Cournapeau wrote: Gael Varoquaux wrote: Obviously, the build part has to be well-tuned for the machinery to work, but there is a lot of value here. Ah yes, setuptools does have this. But this is specific to setuptools, bare distutils does not have this test command, right ? Dunno, sorry. The scale of my ignore of distutils and related subjects would probably impress you :). Gaël, looking forward to your tutorial on scons. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] distutils and inplace build: is numpy supposed to work ?
Hi, I wanted to know if numpy was supposed to work when built in place through the -i option of distutils. The reason why I am asking it that I would like to support it in numscons, and I cannot make it work when using distutils. Importing numpy works in the source tree, but most tests fail because of some missing imports; I have a lots of those: == ERROR: Check that matrix type is preserved. -- Traceback (most recent call last): File /usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py, line 69, in test_matrix_a_and_b self.do(a, b) File /usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py, line 99, in do assert_almost_equal(a, dot(multiply(u, s), vt)) File /usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py, line 22, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File numpy/testing/utils.py, line 171, in assert_almost_equal from numpy.core import ndarray File core/__init__.py, line 27, in module __all__ += numeric.__all__ NameError: name 'numeric' is not defined Is this expected, or am I doing something wrong ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 10:34:04AM -0400, Kevin Jacobs [EMAIL PROTECTED] wrote: The morale of this discussion, for me, is that just because _you_ don't care about a particular aspect or feature, doesn't mean that others don't or shouldn't. Your workarounds may not be viable for me and vice-versa. So let's just go with the spirit of open source and encourage those motivated to controbute to do so, provided their suggestions are sensible and do not break code. I fully agree ehre. And if people improve numpy's startup time with breaking or obfuscating stuff, I am very happy. I was just trying to help :). Yes, the value of open source is that different people improve the same tools to meet different goals, thus we should always keep on open ear to other people's requirements, especially if they come up with high-quality code. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANNOUNCE] Traits 3.0 has been released
On Wed, Jul 30, 2008 at 9:25 PM, Dave Peterson [EMAIL PROTECTED]wrote: Hello, I am very pleased to announce that Traits 3.0 has just been released! All of the URLs on PyPi to Enthought seem to be broken (e.g., http://code.enthought.com/traits). Can you give an example showing how traits work? I'm mildly intrigued, but too lazy to dig beyond the first broken link. Thanks, -Kevin ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANNOUNCE] Traits 3.0 has been released
On Thu, Jul 31, 2008 at 10:38:42AM -0400, Kevin Jacobs [EMAIL PROTECTED] wrote: All of the URLs on PyPi to Enthought seem to be broken (e.g., [2]http://code.enthought.com/traits). Can you give an example showing how traits work? I'm mildly intrigued, but too lazy to dig beyond the first broken link. The proper URL is http://code.enthought.com/projects/traits/ . This has been reported and will be fixed ASAP. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
Andrew Dalke wrote: If I had my way, remove things like (in numpy/__init__.py) import linalg import fft import random import ctypeslib import ma as a side benefit, this might help folks using py2exe, py2app and friends -- as it stands all those sub-modules need to be included in your app bundle regardless of whether they are used. I recall having to explicitly add them by hand, too, though that may have been a matplotlib.numerix issue. but leave the list of submodules in __all__ so that from numpy import * works. Of course, no one should be doing that anyway ;-) And for what it's worth, I've found myself very frustrated by how long it takes to start up python and import numpy. I often do whip out the interpreter to do something fast, and I didn't used to have to wait for it. On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
Christopher Barker wrote: On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy! Hot or cold ? If hot, there is something horribly wrong with your setup. On my macbook, it takes ~ 180 ms to to python -c import numpy, and ~ 100 ms on linux (same machine). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
Stéfan van der Walt wrote: No one is *waiting* for NumPy to start. I am, and probably 10 times, a day, yes. And it's a major issue for CGI, though maybe no one's using that anymore anyway. Just by answering this e-mail I could have (and maybe should have) started NumPy three hundred and sixty times. sure, but I like wasting my time on mailing lists -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
David Cournapeau wrote: Christopher Barker wrote: On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy! Hot or cold ? If hot, there is something horribly wrong with your setup. hot -- it takes about 10 cold. I've been wondering about that. time python -c import numpy real0m8.383s user0m0.320s sys 0m7.805s and similar results if run multiple times in a row. Any idea what could be wrong? I have no clue where to start, though I suppose a complete clean out and re-install of python comes to mind. oh, and this is a dual G5 PPC (which should have a faster disk than your Macbook) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, 31 Jul 2008 10:12:22 -0700 Christopher Barker [EMAIL PROTECTED] wrote: David Cournapeau wrote: Christopher Barker wrote: On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy! Hot or cold ? If hot, there is something horribly wrong with your setup. hot -- it takes about 10 cold. I've been wondering about that. time python -c import numpy real0m8.383s user0m0.320s sys 0m7.805s and similar results if run multiple times in a row. Any idea what could be wrong? I have no clue where to start, though I suppose a complete clean out and re-install of python comes to mind. oh, and this is a dual G5 PPC (which should have a faster disk than your Macbook) -Chris No idea, but for comparison time /usr/bin/python -c import numpy real0m0.295s user0m0.236s sys 0m0.050s [EMAIL PROTECTED]:~/svn/matplotlib cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 10 model name : mobile AMD Athlon (tm) 2500+ stepping: 0 cpu MHz : 662.592 cache size : 512 KB fdiv_bug: no hlt_bug : no f00f_bug: no coma_bug: no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr sse pni syscall mp mmxext 3dnowext 3dnow bogomips: 1316.57 Nils ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 1:12 PM, Christopher Barker [EMAIL PROTECTED]wrote: David Cournapeau wrote: Christopher Barker wrote: On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7 seconds to import numpy! Hot or cold ? If hot, there is something horribly wrong with your setup. hot -- it takes about 10 cold. I've been wondering about that. time python -c import numpy real0m8.383s user0m0.320s sys 0m7.805s and similar results if run multiple times in a row. Any idea what could be wrong? I have no clue where to start, though I suppose a complete clean out and re-install of python comes to mind. Is only 'import numpy' slow, or other packages import slowly too ? Are there remote directories in your pythonpath ? Do you have old `eggs` in the site-packages directory that point to remote directories (installed with setuptools developp) ? Try cleaning the site-packages directory. That did the trick for me once. David oh, and this is a dual G5 PPC (which should have a faster disk than your Macbook) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Numpy 1.1.1 release notes.
Hi All, I've attached draft release notes for Numpy 1.1.1. If you have anything to add or correct, let me know. Chuck Numpy 1.1.1 is a bug fix release featuring major improvements in Python 2.3.x compatibility and masked arrays. Python 2.3.x fixes Robert Kern -- remove developement branch of f2py. Pierre GM -- masked array, fix dictionary update. Pierre GM -- masked array, fix itertools groupby dependency. Pierre GM -- masked array, fix use of iterators. Neil Muller -- fix test dependency on pretty print (#828). General improvements David Cournapeau -- scons build fixes/improvements. David Cournapeau -- ctypes load_library fix. Stefan van der Walt -- Howto document updates. Fernando Perez -- cython updates. Robert Kern -- PyPI metadata fixes. Pierre GM -- masked array, improved support for flexible dtypes. Bug fixes #791 -- Travis Oliphant, std/var not properly calling array_finalize. #798 -- Stefan van der Walt, piecewise exposes raw memory. #810 -- David Huard, Verbose argument not effective for ndarrays in assert_equal. #828 -- Neill Muller, numpy file format format test failures on Solaris. #825 -- Neil Muller/Charles Harris, SIGBUS in UNICODE_getattr on Sparc Linux. #837 -- Pauli Virtan, Infinite loop in fromfile fromstring #841 -- Charles Harris, make numpy complex types print like Python. #843 -- Michael Abbott, reference count error in PyArray_CanCoerceScalar #844 -- David Cournapeau, failure with numpy inner. #848 -- Michael Abbott/Charles Harris/Travis Oliphant, reference leak. #849 -- Michael Abbott, reference to deallocated object. #850 -- Michael Abbott, memory leak. #857 -- Pierre GM, dividing masked array with newaxis fails. --- Robert Kern, make numeric.py functions return booleans. --- Robert Kern, correct old numeric typecodes. --- Robert Kern, fix raising exception in tests. --- Robert Kern, make sure Zipf results are in allowable range. --- Robert Kern, fix c++ style comment. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] The date/time dtype and the casting issue
A Thursday 31 July 2008, Alan G Isaac escrigué: A Thursday 31 July 2008, Matt Knox escrigué: While on the topic of FAME... being a financial analyst, I really am quite fond of the multitude of quarterly frequencies we have in the timeseries package (with different year end points) because they are very useful when doing things like calenderizing earnings from companies with different fiscal year ends. On Thu, 31 Jul 2008, Francesc Alted apparently wrote: Well, introducing a quarter should not be difficult. We just wanted to keep the set of supported time units under a minimum (the list is already quite large). We thought that the quarter fits better as a 'derived' time unit, similarly as biweekly, semester or biyearly (to name just a few examples). However, if quarters are found to be much more important than other derived time units, they can go into the proposal too. Quarterly frequency is probably the most analyzed frequency in macroeconometrics. Widely used macroeconometrics packages (e.g., EViews) traditionally support only three explicit frequencies: annual, quarterly, and monthly. I see. However, I forgot to mention that another reason for not including the quarters is that they normally need flexibility to be defined as starting in *any* month of the year. As we don't wanted to provide an ``origin`` metadata in the proposal (things got too complex already, as you can see in the third proposal that I sent to this list yesterday), then the usefulness of such an 'unflexible' quarters would be rather limited. So, in the end, I think it is best to avoid them for the dtype (and add support for them in the ``Date`` class). Cheers, -- Francesc Alted ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] distutils and inplace build: is numpy supposed to work ?
On Thu, Jul 31, 2008 at 8:18 AM, David Cournapeau [EMAIL PROTECTED] wrote: Hi, I wanted to know if numpy was supposed to work when built in place through the -i option of distutils. The reason why I am asking it that I would like to support it in numscons, and I cannot make it work when using distutils. Importing numpy works in the source tree, but most tests fail because of some missing imports; I have a lots of those: Robert made some fixes to support in place builds, so if it doesn't work, it's probably a bug. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.1.1 release notes.
Thu, 31 Jul 2008 12:12:54 -0600, Charles R Harris wrote: Hi All, I've attached draft release notes for Numpy 1.1.1. If you have anything to add or correct, let me know. [clip] Bug fixes #854, r5456? -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
hot -- it takes about 10 cold. I've been wondering about that. time python -c import numpy real0m8.383s user0m0.320s sys 0m7.805s and similar results if run multiple times in a row. What does python -c import sys; print sys.path say ? Any idea what could be wrong? I have no clue where to start, though I suppose a complete clean out and re-install of python comes to mind. oh, and this is a dual G5 PPC (which should have a faster disk than your Macbook) disk should not matter. If hot, everything should be in the IO buffer, opening a file is of the order of a few micro seconds (that's certainly the order on Linux; the VM on Mac OS X is likely not as good, but still). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.1.1 release notes.
On Thu, Jul 31, 2008 at 12:23 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: Thu, 31 Jul 2008 12:12:54 -0600, Charles R Harris wrote: Hi All, I've attached draft release notes for Numpy 1.1.1. If you have anything to add or correct, let me know. [clip] Bug fixes #854, r5456? Added... Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 10:12:22AM -0700, Christopher Barker wrote: I've been wondering about that. time python -c import numpy real0m8.383s user0m0.320s sys 0m7.805s I don't know what is wrong, but this is plain wrong, unless you are on a distant file system, or something usual. On the box I am currently on, I get: python -c import numpy 0.10s user 0.03s system 101% cpu 0.122 total And this matches my overall experience. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.1.1 release notes.
Chuck, Can you remove the entry Pierre GM -- masked array, improved support for flexible dtypes. from General Improvements ? The work was done for 1.2 and not completely backported, so that's not really a lot of improvements. It will for 1.2, however: when is this one supposed to be released ? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 07:46:20AM -0500, Nathan Bell wrote: On Thu, Jul 31, 2008 at 7:31 AM, Hanni Ali [EMAIL PROTECTED] wrote: I would just to highlight an alternate use of numpy to interactive use. We have a cluster of machines which process tasks on an individual basis where a master tasks may spawn 600 slave tasks to be processed. These tasks are spread across the cluster and processed as scripts in a individual python thread. Although reducing the process time by 300 seconds for the master task is only about a 1.5% speedup (total time can be i excess of 24000s). We process large number of these tasks in any given year and every little helps! There are other components of NumPy/SciPy that are more worthy of optimization. Given that programmer time is a scarce resource, it's more sensible to direct our efforts towards making the other 98.5% of the computation faster. This is true in general, but I have a different use case for one of my programs that uses numpy on a cluster. Basically, the program gets called thousands of times per day and the runtime for each is only a second or two. In this case I am much more dominated by numpy's import time. Scott PS: Yes, I could change the way that the routine works so that it is called many fewer times, however, that would be very difficult (although not impossible). A free speedup due to faster numpy import would be very nice. -- Scott M. RansomAddress: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: [EMAIL PROTECTED] Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 05:43, Andrew Dalke [EMAIL PROTECTED] wrote: On Jul 31, 2008, at 12:03 PM, Robert Kern wrote: But you still can't remove them since they are being used inside numerictypes. That's why I labeled them internal utility functions instead of leaving them with minimal docstrings such that you would have to guess. My proposal is to replace that code with a table mapping the type name to the uppercase/lowercase/capitalized forms, thus eliminating the (small) amount of time needed to import string. It makes adding new types slightly more difficult. I know it's a tradeoff. Probably not a bad one. Write up the patch, and then we'll see how much it affects the import time. I would much rather that we discuss concrete changes like this rather than rehash the justifications of old decisions. Regardless of the merits about the old decisions (and I agreed with your position at the time), it's a pointless and irrelevant conversation. The decisions were made, and now we have a user base to whom we have promised not to break their code so egregiously again. The relevant conversation is what changes we can make now. Some general guidelines: 1) Everything exposed by from numpy import * still needs to work. a) The layout of everything under numpy.core is an implementation detail. b) _underscored functions and explicitly labeled internal functions can probably be modified. c) Ask about specific functions when in doubt. 2) The improvement in import times should be substantial. Feel free to bundle up the optimizations for consideration. 3) Moving imports from module-level down into the functions where they are used is generally okay if we get a reasonable win from it. The local imports should be commented, explaining that they are made local in order to improve the import times. 4) __import__ hacks are off the table. 5) Proxy objects ... I would really like to avoid proxy objects. They have caused fragility in the past. 6) I'm not a fan of having environment variables control the way numpy gets imported, but I'm willing to consider it. For example, I might go for having proxy objects for linalg et al. *only* if a particular environment variable were set. But there had better be a very large improvement in import times. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] distutils and inplace build: is numpy supposed to work ?
On Thu, Jul 31, 2008 at 09:18, David Cournapeau
[EMAIL PROTECTED] wrote:
Hi,
I wanted to know if numpy was supposed to work when built in place
through the -i option of distutils. The reason why I am asking it that I
would like to support it in numscons, and I cannot make it work when
using distutils. Importing numpy works in the source tree, but most
tests fail because of some missing imports; I have a lots of those:
==
ERROR: Check that matrix type is preserved.
--
Traceback (most recent call last):
File
/usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py, line
69, in test_matrix_a_and_b
self.do(a, b)
File
/usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py, line
99, in do
assert_almost_equal(a, dot(multiply(u, s), vt))
File
/usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py, line
22, in assert_almost_equal
old_assert_almost_equal(a, b, decimal=decimal, **kw)
File numpy/testing/utils.py, line 171, in assert_almost_equal
from numpy.core import ndarray
File core/__init__.py, line 27, in module
__all__ += numeric.__all__
NameError: name 'numeric' is not defined
Is this expected, or am I doing something wrong ?
I have been running numpy built inplace for a long time now. As far as
I can tell, this only shows up when running numpy.test() while in the
numpy trunk checkout directory. I think it's an interaction with the
way nose traverses packages to locate tests. numpy/core/__init__.py is
a bit odd; it does from numeric import * and expects numeric to
then be in the namespace. I think this only happens when the import
machinery knows that it's in a package. nose uses __import__() when
scouting around the package, so it misses this. For example,
[~]$ ls foo
__init__.py bar.py
[~]$ cat foo/__init__.py
from bar import x
print dir()
print bar.x
[~]$ cat foo/bar.py
x = 1
[~]$ python -c import foo
['__builtins__', '__doc__', '__file__', '__name__', '__path__', 'bar', 'x']
1
[~]$ python -c __import__('foo.__init__')
['__builtins__', '__doc__', '__file__', '__name__', '__path__', 'bar', 'x']
1
['__builtins__', '__doc__', '__file__', '__name__', 'x']
Traceback (most recent call last):
File string, line 1, in module
File foo/__init__.py, line 3, in module
print bar.x
NameError: name 'bar' is not defined
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Fri, Aug 1, 2008 at 5:02 AM, Robert Kern [EMAIL PROTECTED] wrote: 5) Proxy objects ... I would really like to avoid proxy objects. They have caused fragility in the past. One recurrent problem around import times optimization is that it is some work to improve it, but it takes one line to destroy it all. For example, inspect import came back, and this alone is ~10-15 % of my import time on mac os x (from ~ 180 to ~160). This would be the main advantage of lazy import; but does it really worth the trouble, since it brings some complexity as you mentionned last time we had this discussion ? Maybe a simple test script to check for known costly import would be enough (running from time to time ?). Maybe ctypes can be loaded in the fly, too. Those are the two obvious hotspot ( ~ 25 % altogether). with a recent SVN checkout 6) I'm not a fan of having environment variables control the way numpy gets imported, but I'm willing to consider it. For example, I might go for having proxy objects for linalg et al. *only* if a particular environment variable were set. But there had better be a very large improvement in import times. linalg does not seem to have a huge impact. It is typically much faster to load than ctypeslib or inspect. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] patches to reduce import costs
I don't see a place to submit patches. Is there a patch manager for numpy? Here's a patch to defer importing 'tempfile' until needed. I previously mentioned one other place that didn't need tempfile. With this there is no 'import tempfile' during 'import numpy' This improves startup by about 7% --- numpy/lib/_datasource.py(revision 5576) +++ numpy/lib/_datasource.py(working copy) @@ -35,7 +35,6 @@ __docformat__ = restructuredtext en import os -import tempfile from shutil import rmtree from urlparse import urlparse @@ -131,6 +130,7 @@ self._destpath = os.path.abspath(destpath) self._istmpdest = False else: +import tempfile self._destpath = tempfile.mkdtemp() self._istmpdest = True Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
