[Numpy-discussion] REMINDER: trunk is about to be frozen for 1.4.0
Hi, A quick remainder: the trunk will be closed for 1.4.0 changes within a few hours. After that time, the trunk should only contain things which will be in 1.5.0, and the 1.4.0 changes will be in the 1.4.0 branch, which should contain only bug fixes. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Initial implementation of histogram_discrete()
On 11/15/2009 01:16 AM, David Warde-Farley wrote: On 14-Nov-09, at 10:57 AM, Bruce Southey wrote: Is it just that bincount does not count negative numbers? If so, then I would strongly argue that is insufficient for creating new function. Rather you need to provide a suitable patch to fix bincount or replace bincount with a better version. The whole point of bincount is that in the result array, the indices correspond to _values_ in the original array, i.e. bc = bincount(arr) then bc[3] gives me the number of 3's that appear in arr. This is lost if bincount starts reporting negatives. I suppose you could return a tuple of two arrays, optionally, but I don't know how much use it would get. Isn't this indexing invalid if you have a negative value in the original array in the first place? While I have not tried Priit's code (as it is in C), the provided output with negative values does not appear to maintain this property. Really without changing the output type, I do not see how you can keep this indexing property with negative values. So really the comparison has to be with histogram(). If you really need to do bincount on an array with negative ints, you can just add -arr.min() to the array first and subtract the same value when done. Assuming you don't have a huge array this will be very fast, and can even be done in place. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Josef provided the histogram solution using np.arange for the bins that appears to be handle negative values except that the indexing is refers to position not the value. That is the index is value minus the minimum of (zero, min(array)) - which is the indexing as required by Priit's code. Bruce ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 38, Issue 52
Jake VanderPlas wrote: It sounds like all of this could be done very simply without going to C, using a class based on numpy.ndarray. The following works for 1D arrays, behaves like a regular 1D numpy array, and could be easily improved with a little care. Is this what you had in mind? import numpy #easy scalable array class class scarray: def __init__(self,*args,**kwargs): self.__data = numpy.ndarray(*args,**kwargs) def append(self,val): tmp = self.__data self.__data = numpy.ndarray(tmp.size+1) self.__data[:-1] = tmp self.__data[-1] = val del tmp def __getattr__(self,attr): return getattr(self.__data,attr) The problem here is that it's re-allocating memory with every single addition of an element. It's pretty common to pre-allocate some extra space for this kind of thing (python lists, std vectors, etc, etc, etc), so I assumed that it would be performance killer. However, you know what they say about premature optimization, so a test or two is in order. This is incrementally adding 1 integers, one point at a time: Using the suggested code: In [21]: timeit p.scarray1(1) 10 loops, best of 3: 1.71 s per loop Using my accumulator code: In [23]: timeit p.accum1(1) 10 loops, best of 3: 25.6 ms per loop So all that memory re-allocation really does kill performance. In [24]: timeit p.list1(1) 100 loops, best of 3: 9.96 ms per loop But, of course, lists are still faster. I think this is because I'm adding python integers, which are already python objects, so that's exactly what lists are for -- you can't beat them. This wouldn't apply to using them from C, however. Also, I see a change when we add chunks of data already in a numpy array, with .extend(): # adding a sequence of ten integers at a time: In [40]: timeit profile_accumulator.accum_extend1(1) 100 loops, best of 3: 6.36 ms per loop In [41]: timeit profile_accumulator.accum_extend1(1) 100 loops, best of 3: 6.22 ms per loop # about the same speed # but when I add 100 elements at a time: In [46]: timeit profile_accumulator.list_extend1(1) 10 loops, best of 3: 56.6 ms per loop In [47]: timeit profile_accumulator.accum_extend1(1) 100 loops, best of 3: 13.3 ms per loop # I start to see a real advantage to the numpy accumulator approach. Is # that a real use-case? I'm not sure. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 38, Issue 52
oops, I meant to include my code with that last note. Here it is. accumulator.py: is my implementation. easy_scale.py: is Jake's suggested implementation. profile_accumulator.py: contains some test functions that can be run with ipython's timeit test_accumulator.py: is test code that can be run with nosetests. Any comments, suggestions, etc. welcome. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov accumulator.py Description: application/python easy_scale.py Description: application/python profile_accumulator.py Description: application/python test_accumulator.py Description: application/python ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
Charles R Harris wrote: I would like some advise on the best way to add the new functions. I've added a new package polynomial, and that package contains four new modules: chebyshev, polynomial, polytemplate, polyutils. This seems to belong more in scipy than numpy, but I'll leave that to others to decide. whether or not to include all of the functions in these packages in the __init__.py, or to just import the modules. Are any of them compiled code? I've been very frustrated when I can't use some pure python stuff in scipy because broken compiled fortran extensions are getting imported that I don't even need. If that isn't an issue, and the polynomial package would end up with only a handful of names, then I say import them all. Another way to ask this: would there by ANY names in the polynomial package if you don't import the modules? If there is compiled code, the import could fail gracefully, and then you could still pull it all in. OTOH, what this does is bring stuff into memory unnecessarily, and also brings it into stand-alone bundles (py2exe, py2app, etc). So if these modules are not small, then it's probably better to have to import them explicitly. Also -- do you foresee many more polynomial types in the future? I know I'd like to see Hermite. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
On Mon, Nov 16, 2009 at 10:43 AM, Christopher Barker chris.bar...@noaa.govwrote: Charles R Harris wrote: I would like some advise on the best way to add the new functions. I've added a new package polynomial, and that package contains four new modules: chebyshev, polynomial, polytemplate, polyutils. This seems to belong more in scipy than numpy, but I'll leave that to others to decide. whether or not to include all of the functions in these packages in the __init__.py, or to just import the modules. Are any of them compiled code? I've been very frustrated when I can't use some pure python stuff in scipy because broken compiled fortran extensions are getting imported that I don't even need. It's all python. If that isn't an issue, and the polynomial package would end up with only a handful of names, then I say import them all. Another way to ask this: would there by ANY names in the polynomial package if you don't import the modules? That's what I ended up doing. You still need to do import numpy.polynomial to get to them, they aren't automatically imported into the numpy namespace. If there is compiled code, the import could fail gracefully, and then you could still pull it all in. OTOH, what this does is bring stuff into memory unnecessarily, and also brings it into stand-alone bundles (py2exe, py2app, etc). So if these modules are not small, then it's probably better to have to import them explicitly. Also -- do you foresee many more polynomial types in the future? I know I'd like to see Hermite. Been thinking about it. Division/multiplication can get hard, but a more restricted set of operations -- division/multiplication by x -- would cover most of the common uses. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] failure building trunk with mingw
On Mon, Nov 16, 2009 at 12:48 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: josef.p...@gmail.com wrote: Are there new changes to the configuration needed? mingw 3.4.5, WindowsXP, Python 2.5.2 Python.h is not picked up anymore: _configtest.c:1:20: Python.h: No such file or directory Josef C:\Josef\_progs\Subversion\numpy-trunksetup.py bdist Could you try with python setup.py build -c mingw32 bdist I have just tried on the same configuration as you, and I cannot reproduce the problem with the above command. David It was a change I did. Following a recommendation on the cython list I had added include_dirs in distutils.cfg [build_ext] include_dirs=c:\programs\python25\lib\site-packages\numpy\core\include As a consequence numpy build had dropped the python include directory from the build options. Now, the numpy build runs for a while then breaks while building umath. Any ideas? setup.py bdist and python setup.py build -c mingw32 bdist give the same results. Thanks, Josef ... building 'numpy.core.umath' extension compiling C sources C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes creating build\temp.win32-2.5\Release\numpy\core\src\umath compile options: '-Ibuild\src.win32-2.5\numpy\core\src\umath -Inumpy\core\includ e -Ibuild\src.win32-2.5\numpy\core\include/numpy -Inumpy\core\src\private -Inump y\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -In umpy\core\src\umath -Inumpy\core\include -IC:\Programs\Python25\include -IC:\Pro grams\Python25\PC -Ibuild\src.win32-2.5\numpy\core\src\multiarray -Ibuild\src.wi n32-2.5\numpy\core\src\umath -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Ibuild\src.win32-2.5\numpy\core\s rc\umath -Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core\include/numpy -In umpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -In umpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\include -IC:\Progra ms\Python25\include -IC:\Programs\Python25\PC -Ibuild\src.win32-2.5\numpy\core\s rc\multiarray -Ibuild\src.win32-2.5\numpy\core\src\umath -c numpy\core\src\umath \umathmodule_onefile.c -o build\temp.win32-2.5\Release\numpy\core\src\umath\umat hmodule_onefile.o g++ -mno-cygwin -shared build\temp.win32-2.5\Release\numpy\core\src\umath\umathm odule_onefile.o -LC:\Programs\Python25\libs -LC:\Programs\Python25\PCBuild -Lbui ld\temp.win32-2.5 -lnpymath -lpython25 -lmsvcr71 -o build\lib.win32-2.5\numpy\co re\umath.pyd build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xce07): undefined reference to `npy_spacingf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xcefc): undefined reference to `npy_nextafterf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xdeb7): undefined reference to `npy_spacing' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xdfae): undefined reference to `npy_nextafter' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xef75): undefined reference to `npy_spacingl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xf0b0): undefined reference to `npy_nextafterl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c3c6): undefined reference to `npy_csqrtf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c436): undefined reference to `npy_clogf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c4d6): undefined reference to `npy_cexpf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c78c): undefined reference to `npy_cpowf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d28b): undefined reference to `npy_csqrt' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d30b): undefined reference to `npy_clog' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d3ab): undefined reference to `npy_cexp' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d711): undefined reference to `npy_cpow' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e379): undefined reference to `npy_csqrtl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e429): undefined reference to `npy_clogl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e4e9): undefined reference to `npy_cexpl'
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
2009/11/16 Christopher Barker chris.bar...@noaa.gov: Charles R Harris wrote: I would like some advise on the best way to add the new functions. I've added a new package polynomial, and that package contains four new modules: chebyshev, polynomial, polytemplate, polyutils. This seems to belong more in scipy than numpy, but I'll leave that to others to decide. whether or not to include all of the functions in these packages in the __init__.py, or to just import the modules. Are any of them compiled code? I've been very frustrated when I can't use some pure python stuff in scipy because broken compiled fortran extensions are getting imported that I don't even need. If that isn't an issue, and the polynomial package would end up with only a handful of names, then I say import them all. Another way to ask this: would there by ANY names in the polynomial package if you don't import the modules? If there is compiled code, the import could fail gracefully, and then you could still pull it all in. OTOH, what this does is bring stuff into memory unnecessarily, and also brings it into stand-alone bundles (py2exe, py2app, etc). So if these modules are not small, then it's probably better to have to import them explicitly. Also -- do you foresee many more polynomial types in the future? I know I'd like to see Hermite. I have kind of gone silent on this issue, in part because I have been busy with other things. I think that Charles Harris' framework for working with polynomials is perfectly sufficient for adding Chebyshev polynomials, and since they're finished and working and fill a concrete need, they should probably go into numpy or scipy as is. But if you want to start introducing other types of polynomials - Hermite, Lagrange interpolation based, Bernstein based, or other - I think we would need to revive the discussion about how to unify all these different types. Anne -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
On Mon, Nov 16, 2009 at 3:06 PM, Anne Archibald peridot.face...@gmail.comwrote: 2009/11/16 Christopher Barker chris.bar...@noaa.gov: Charles R Harris wrote: I would like some advise on the best way to add the new functions. I've added a new package polynomial, and that package contains four new modules: chebyshev, polynomial, polytemplate, polyutils. This seems to belong more in scipy than numpy, but I'll leave that to others to decide. whether or not to include all of the functions in these packages in the __init__.py, or to just import the modules. Are any of them compiled code? I've been very frustrated when I can't use some pure python stuff in scipy because broken compiled fortran extensions are getting imported that I don't even need. If that isn't an issue, and the polynomial package would end up with only a handful of names, then I say import them all. Another way to ask this: would there by ANY names in the polynomial package if you don't import the modules? If there is compiled code, the import could fail gracefully, and then you could still pull it all in. OTOH, what this does is bring stuff into memory unnecessarily, and also brings it into stand-alone bundles (py2exe, py2app, etc). So if these modules are not small, then it's probably better to have to import them explicitly. Also -- do you foresee many more polynomial types in the future? I know I'd like to see Hermite. I have kind of gone silent on this issue, in part because I have been busy with other things. I think that Charles Harris' framework for working with polynomials is perfectly sufficient for adding Chebyshev polynomials, and since they're finished and working and fill a concrete need, they should probably go into numpy or scipy as is. But if you want to start introducing other types of polynomials - Hermite, Lagrange interpolation based, Bernstein based, or other - I think we would need to revive the discussion about how to unify all these different types. Yes, that still needs thinking about. The recursion for the Chebyshev polynomials (and powers) has constant coefficients apart from the x, and that makes a lot of simplifications possible. In particular, the resulting polynomials can be represented as constant combinations of powers of the two roots of the recursion relation. That simplifies a lot of things that can be difficult in the more general case. And that isn't even getting started on non-graded things like the Bernstein and Lagrange interpolation based schemes. There is a lot of interesting combinatorial work going on in the polynomial business that I am not familiar with. Some of that may be of use to us in looking for a framework. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
Charles R Harris wrote: That's what I ended up doing. You still need to do import numpy.polynomial to get to them, they aren't automatically imported into the numpy namespace. good start. This brings up a semi-off-topic question: Is there a way to avoid importing everything when importing a module deep in a big package? My example: we just started using scipy.spatial.ckdtree in a project that doesn't (yet) use anything else in scipy. The import line: from scipy.spatial.ckdtree import cKDTree takes a LONG time and adds 262 modules to sys.modules It also added 15MB to our py2exe package and who knows how much memory it is using up. As far as I can tell, it is perfectly functional with only ckdtree.so and kdtree.py Is there a solution to this short of a complete re-structuring of scipy? -Chris Tests: Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type help, copyright, credits or license for more information. mem use: 37404 import sys len(sys.modules) 29 import numpy len(sys.modules) 174 mem use: 45700 len(sys.modules) 291 mem use: 52384 So I guess memory use isn't that bad (by modern standards!), but I'd still rather not have 115 or so modules in there! -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
On Mon, Nov 16, 2009 at 18:05, Christopher Barker chris.bar...@noaa.gov wrote: Charles R Harris wrote: That's what I ended up doing. You still need to do import numpy.polynomial to get to them, they aren't automatically imported into the numpy namespace. good start. This brings up a semi-off-topic question: Is there a way to avoid importing everything when importing a module deep in a big package? The package authors need to keep the __init__.py files clear. There is nothing you can do as a user. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] finding close together points.
2009/11/16 Christopher Barker chris.bar...@noaa.gov: Anne Archibald wrote: 2009/11/13 Christopher Barker chris.bar...@noaa.gov: Wow! great -- you sounded interested, but I had no idea you'd run out and do it! thanks! we'll check it out. well, it turns out the Python version is unacceptably slow. However, we found we could use ckdtree, and use: tree.query(points, 2, # Number of points to return per point given. distance_upper_bound = distance_tolerance, ) This gives us a set of pairs of points closer than our distance tolerance -- it includes duplicates, of course, but even when we filter those in pure python, it's is nice and speedy. It looks like it would be pretty easy to add this to the Cython code, but it's working now for us, so... You probably noticed this, but I should remind you that if a point has more than one nearby neighbor, this may not give you what you wanted. In particular, if there are three points all within the fixed distance from each other, you may not find all three pairs this way. Anne Thanks again, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
2009/11/16 Robert Kern robert.k...@gmail.com: On Mon, Nov 16, 2009 at 18:05, Christopher Barker chris.bar...@noaa.gov wrote: Charles R Harris wrote: That's what I ended up doing. You still need to do import numpy.polynomial to get to them, they aren't automatically imported into the numpy namespace. good start. This brings up a semi-off-topic question: Is there a way to avoid importing everything when importing a module deep in a big package? The package authors need to keep the __init__.py files clear. There is nothing you can do as a user. The reason numpy and scipy don't do this is largely historical - Numeric had a nearly flat namespace, and imported all the submodules in any case, and numpy is trying to remain somewhat compatible; scipy originally tried to reexport all the numpy symbols, for some reason, and there too it is maintaining compatibility. Since spatial is new, though, it should be pretty good about not inflating your imports unnecessarily. It does, of course, import various things itself, which may balloon out the imports. Anne -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
Robert Kern wrote: Is there a way to avoid importing everything when importing a module deep in a big package? The package authors need to keep the __init__.py files clear. There is nothing you can do as a user. I figured. so, to bring this back on-topic: I recommend that no package imports anything it doesn't need to. It's just not that hard to import what you need. numpy appears to be pretty bad in this regard: len(sys.modules) 29 import numpy len(sys.modules) 174 with a simple import numpy, I get all kinds of stuff I'm unlikely to need in every app, and if I do, I import it anyway: numpy.testing numpy.ma numpy.random numpy.linalg numpy.lib.financial numpy.lib.polynomial numpy.fft It's probably way too late from a backward compatibility perspective to change any of this, but I'd love to see that cleaned up. Maybe we can clean up scipy, though? We could have a two-API apporach, ike what MPL has been doing. One that import every darn thing for ease of us on the command line, and one that only imports what you ask for. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
Anne Archibald wrote: The reason numpy and scipy don't do this is largely historical - Numeric had a nearly flat namespace, I know. Despite namespaces being one honking great idea, it seems to have taken a while to catch on. Since spatial is new, though, it should be pretty good about not inflating your imports unnecessarily. well, maybe -- it seems to import everything when you do an import scipy.spatial Since kdtree is about all that's there, it's not a big deal, but much like the polynomial namespace, if we envision a day when there may be all sorts of lovely spatial indexing options in there, maybe it shouldn't bring them all into the spatial namespace. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the new polynomial/chebyshev modules.
On Mon, Nov 16, 2009 at 18:39, Christopher Barker chris.bar...@noaa.gov wrote: Robert Kern wrote: Is there a way to avoid importing everything when importing a module deep in a big package? The package authors need to keep the __init__.py files clear. There is nothing you can do as a user. I figured. so, to bring this back on-topic: I recommend that no package imports anything it doesn't need to. It's just not that hard to import what you need. numpy appears to be pretty bad in this regard: len(sys.modules) 29 import numpy len(sys.modules) 174 It's not *quite* as bad as that. There are a lot of keys in sys.modules that point to None as an optimization (e.g. if foo/bar.py imports math, let's say, the import system will look up foo.math, which fails, so sys.modules['foo.math'] = None so it doesn't try that again). Here is the correct way to measure that: import sys old = set(sys.modules) len(old) 30 import numpy new = set(sys.modules) from_numpy = [m for m in (new - old) if sys.modules[m] is not None] len(from_numpy) 93 from_numpy ['numpy.lib._iotools', 'numpy.core.info', 'numpy.lib.getlimits', 'ctypes._endian', 'numpy.core.numerictypes', 'struct', 'numpy.random.info', 'numpy.linalg', 'numpy.testing', 'numpy.core.umath', 'numpy.core.scalarmath', 'string', 'numpy.lib.arraysetops', 'numpy.version', 'numpy.lib.type_check', 'numpy.lib._datasource', 'numpy.lib.io', 'numpy.ma.extras', 'token', 'numpy.fft.fftpack_lite', 'numpy.core.multiarray', 'dis', 'cStringIO', 'numpy.ma.core', 'numpy.add_newdocs', 'numpy.testing.decorators', 're', 'numpy.lib._compiled_base', 'numpy.random.mtrand', 'math', 'numpy.fft.helper', 'inspect', '_ctypes', 'numpy.lib.ufunclike', 'numpy.lib.info', 'ctypes', 'numpy.core._sort', 'numpy.core.memmap', 'traceback', 'itertools', 'numpy.fft.fftpack', 'opcode', 'numpy.linalg.lapack_lite', '__future__', '_sre', 'unittest', 'numpy.random', 'numpy.lib.twodim_base', 'operator', 'sre_constants', 'numpy.lib.arrayterator', 'numpy.lib.financial', 'imp', 'numpy.core.arrayprint', 'tokenize', 'numpy.lib.stride_tricks', 'numpy', 'numpy.core.defmatrix', 'cPickle', 'numpy.ma', 'numpy.testing.utils', 'gestalt', '_struct', 'numpy.core.fromnumeric', 'numpy.ctypeslib', 'numpy.lib.scimath', 'numpy.fft', 'numpy.lib', 'numpy.lib.function_base', 'sre_parse', 'sre_compile', 'numpy.lib.shape_base', 'numpy.lib.polynomial', 'numpy._import_tools', 'numpy.fft.info', 'numpy.core.records', 'numpy.core._dotblas', 'shutil', 'strop', 'numpy.testing.numpytest', 'numpy.core.numeric', 'numpy.linalg.info', 'numpy.core._internal', 'numpy.__config__', 'numpy.core', 'numpy.lib.index_tricks', 'numpy.lib.utils', 'numpy.core.defchararray', 'numpy.lib.format', 'numpy.testing.nosetester', 'time', 'numpy.lib.machar', 'numpy.linalg.linalg'] with a simple import numpy, I get all kinds of stuff I'm unlikely to need in every app, and if I do, I import it anyway: numpy.testing numpy.ma numpy.random numpy.linalg numpy.lib.financial numpy.lib.polynomial numpy.fft It's probably way too late from a backward compatibility perspective to change any of this, but I'd love to see that cleaned up. Oh, good heavens, yes. Maybe we can clean up scipy, though? Some of the parts checked in since 0.7, but otherwise not really, no. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] failure building trunk with mingw
On Tue, Nov 17, 2009 at 3:33 AM, josef.p...@gmail.com wrote: Now, the numpy build runs for a while then breaks while building umath. Any ideas? The behavior of distutils with config files is mysterious, I gave up trying to understand it long ago :) I use scripts instead to control everything from command line. creating build\temp.win32-2.5\Release\numpy\core\src\umath compile options: '-Ibuild\src.win32-2.5\numpy\core\src\umath -Inumpy\core\includ e -Ibuild\src.win32-2.5\numpy\core\include/numpy -Inumpy\core\src\private -Inump y\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -In umpy\core\src\umath -Inumpy\core\include -IC:\Programs\Python25\include -IC:\Pro grams\Python25\PC -Ibuild\src.win32-2.5\numpy\core\src\multiarray -Ibuild\src.wi n32-2.5\numpy\core\src\umath -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Ibuild\src.win32-2.5\numpy\core\s rc\umath -Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core\include/numpy -In umpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -In umpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\include -IC:\Progra ms\Python25\include -IC:\Programs\Python25\PC -Ibuild\src.win32-2.5\numpy\core\s rc\multiarray -Ibuild\src.win32-2.5\numpy\core\src\umath -c numpy\core\src\umath \umathmodule_onefile.c -o build\temp.win32-2.5\Release\numpy\core\src\umath\umat hmodule_onefile.o g++ -mno-cygwin -shared build\temp.win32-2.5\Release\numpy\core\src\umath\umathm odule_onefile.o -LC:\Programs\Python25\libs -LC:\Programs\Python25\PCBuild -Lbui ld\temp.win32-2.5 -lnpymath -lpython25 -lmsvcr71 -o build\lib.win32-2.5\numpy\co re\umath.pyd build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xce07): undefined reference to `npy_spacingf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xcefc): undefined reference to `npy_nextafterf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xdeb7): undefined reference to `npy_spacing' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xdfae): undefined reference to `npy_nextafter' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xef75): undefined reference to `npy_spacingl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xf0b0): undefined reference to `npy_nextafterl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c3c6): undefined reference to `npy_csqrtf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c436): undefined reference to `npy_clogf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c4d6): undefined reference to `npy_cexpf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c78c): undefined reference to `npy_cpowf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d28b): undefined reference to `npy_csqrt' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d30b): undefined reference to `npy_clog' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d3ab): undefined reference to `npy_cexp' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d711): undefined reference to `npy_cpow' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e379): undefined reference to `npy_csqrtl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e429): undefined reference to `npy_clogl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e4e9): undefined reference to `npy_cexpl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1ea15): undefined reference to `npy_cpowl' collect2: ld returned 1 exit status error: Command g++ -mno-cygwin -shared build\temp.win32-2.5\Release\numpy\core\ src\umath\umathmodule_onefile.o -LC:\Programs\Python25\libs -LC:\Programs\Python 25\PCBuild -Lbuild\temp.win32-2.5 -lnpymath -lpython25 -lmsvcr71 -o build\lib.wi n32-2.5\numpy\core\umath.pyd failed with exit status 1 All those refer to recently added functions, which suggest some old files/configuration are used. Be sure to clean the working tree and the build directory and try again. I have just tested it on both wine and a xp vm with mingw, and both build
[Numpy-discussion] Fitting a curve on a log-normal distributed data
Hello, I have a data which represents aerosol size distribution in between 0.1 to 3.0 micrometer ranges. I would like extrapolate the lower size down to 10 nm. The data in this context is log-normally distributed. Therefore I am looking a way to fit a log-normal curve onto my data. Could you please give me some pointers to solve this problem? Thank you. -- Gökhan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fitting a curve on a log-normal distributed data
Theory wise: -Do a linear regression on your data. -Apply a logrithmic transform to your data's dependent variable, and do another linear regression. -Apply a logrithmic transform to your data's independent variable, and do another linear regression. -Take the best regression (highest r^2 value) and execute a back transform. Then, to get your desired extrapolation, simply substitute in the size for the independent variable to get the expected value. If, however, you're looking for how to implement this in NumPy or SciPy, I can't really help :-P Ian ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] min bug
I'm pretty sure this shouldn't happen: In [1]: from numpy import min In [2]: min(5000, 4) Out[2]: 5000 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] min bug
Known issue, I think someone posted about it a while ago too. The numpy min is array aware, and it expects an array. The second argument is the axis, which in the case of a single number doesn't matter. On Tue, 2009-11-17 at 07:07 +, Chris wrote: I'm pretty sure this shouldn't happen: In [1]: from numpy import min In [2]: min(5000, 4) Out[2]: 5000 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] min bug
On Mon, Nov 16, 2009 at 11:07 PM, Chris fonnesb...@gmail.com wrote: I'm pretty sure this shouldn't happen: In [1]: from numpy import min In [2]: min(5000, 4) Out[2]: 5000 The way you're calling it is working like this: min((5000,) , axis=4) so you'd need to do this instead: min((5000,4)) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] min bug
Sebastian Berg wrote: Known issue, I think someone posted about it a while ago too. The numpy min is array aware, and it expects an array. The second argument is the axis, which in the case of a single number doesn't matter. On Tue, 2009-11-17 at 07:07 +, Chris wrote: I'm pretty sure this shouldn't happen: In [1]: from numpy import min In [2]: min(5000, 4) Out[2]: 5000 I think I have to agree with the original poster. It would be more correct to rise an exception because the axis is beyond the number of axes than to return a confusing result. Armando ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] min bug
On Mon, Nov 16, 2009 at 11:34 PM, V. Armando Solé s...@esrf.fr wrote: Sebastian Berg wrote: Known issue, I think someone posted about it a while ago too. The numpy min is array aware, and it expects an array. The second argument is the axis, which in the case of a single number doesn't matter. I think I have to agree with the original poster. It would be more correct to rise an exception because the axis is beyond the number of axes than to return a confusing result. Hm, now that I actually try my example min((5000,), 4) it fails with an axis out of bounds error. I presume there's a reason why a 0-D array gets special treatment? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
