Re: [Numpy-discussion] 32bit integers as list indices
Neilen Marais wrote: Hi, I'm not sure if the following is expected to work on a 64bit machine: In [381]: import numpy as N In [382]: l = range(3) In [383]: i32 = N.array([0,2], N.int32) In [384]: i64 = N.array([0,2], N.int64) In [385]: l[i32[0]] --- exceptions.TypeError Traceback (most recent call last) /home/brick/akademie/NewCode/working/ipython console TypeError: list indices must be integers In [386]: l[i64[0]] Out[386]: 0 I'd expect the 32-bit indices to work since they can be upcast to 64bit without loss. Am I silly for thinking this way, or is it something numpy can/should address? This came up while working with sparse matrices: http://projects.scipy.org/scipy/scipy/ticket/307 It's addressed with Python 2.5 We can't do anything about it for Python 2.4 -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] mysql - record array
John Hunter wrote: John == John Hunter [EMAIL PROTECTED] writes: Erin == Erin Sheldon [EMAIL PROTECTED] writes: Erin The question I have been asking myself is what is the Erin advantage of such an approach?. It would be faster, but by John In the use case that prompted this message, the pull from John mysql took almost 3 seconds, and the conversion from lists John to numpy arrays took more that 4 seconds. We have a list of John about 50 2 tuples of floats. John Digging in a little bit, we found that numpy is about 3x John slower than Numeric here John peds-pc311:~ python test.py with dtype: 4.25 elapsed John seconds w/o dtype 5.79 elapsed seconds Numeric 1.58 elapsed John seconds 24.0b2 1.0.1.dev3432 John Hmm... So maybe the question is -- is there some low hanging John fruit here to get numpy speeds up? And for reference, numarray is 5 times faster than Numeric here and 15 times faster than numpy peds-pc311:~ python test.py with dtype: 4.20 elapsed seconds w/o dtype 5.71 elapsed seconds Numeric 1.60 elapsed seconds numarray 0.30 elapsed seconds 24.0b2 1.0.1.dev3432 1.5.1 import numarray tnow = time.time() y = numarray.array(x, numarray.Float) tdone = time.time() This sounds like it could be definitely be sped up, then. Assign_Array is the relevant code (it then calls PySequence_SetItem) so that basically for k in a.shape[0]: a[k] = x[k] is what is being done. Thus, it might be the indexing code that is causing this to be a slower operation. We should look at what numarray is doing --- it has provided important speed-ups in the past. I still don't have time to look at this, but please file a ticket as we should fix this one. Reference the faster numarray implementation. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Scholarly contribution of NumPy : request for review
I apologize for this bit of spam. If there is anyone on this list (preferrably of Associate Professor or similar rank) who can provide an honest assessment of the scholarly contribution of NumPy and/or SciPy (and has not already done so)? If you can do this and don't mind me quoting you to a tenure-review committee, could you please email me your assessment by Friday morning (Nov. 10). Please include information about your position in the mail. I welcome all assessments (both positive and negative). Thank you very much. Travis Oliphant Assistant Professor Brigham Young University [EMAIL PROTECTED] 801-422-3108 - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] I won't be as visible for a few weeks
I'm writing this to indicate that I won't be as visible on the lists for a few weeks as I have some pressing matters to attend to that will take more of my time. I apologize for not being able to help more right now. Best regards, -Travis Oliphant - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy book
Note that this is not a request to Travis to send me the latest version by private email. That would be inefficient and my need is not that urgent. Nevertheless I think that issue should be settled. There will be an update, soon. I'm currently working on the index, corrections, and formatting issues. The update will be sent in conjunction with the release of 1.0.1 which I am targetting in 2 weeks. Thanks for everybody's patience. If you need an update now, I'm happy to send it to you by private email. Best regards, -Travis O. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Argmax and max inconsistent for object types
Jonathan Wang wrote: On numpy version 1.0, argmax and max give inconsistent results for an array of objects. I've seen this problem in both Python native datetime and mx.DateTime types: There is a bug in argmax for OBJECT arrays in 1.0 (it's fixed in SVN and will be in 1.0.1) -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading records from file and sorting
George Sakkis wrote: Charles R Harris wrote: On 11/1/06, George Sakkis [EMAIL PROTECTED] wrote: Albert Strasheim wrote: Check the thread Strange results when sorting array with fields from about a week back. Travis made some changes to sorting in the presence of fields that should solve your problem, assuming your fields appear in the order you want to sort (i.e. you want to sort f1, f2, f3 and not something like f1, f3, f2). I'm afraid this won't help in my case; I want to sort twice, once by f1 and once by f2. I guess I could make a second file with the fields swapped but this seems more messy and inefficient than Francesc's suggestion. Do you actually want the two different orders, or do you want to sort on the first field, then sort all the items with the same first field on the second field? The former. Sorting on a particular field in-place would be possible if there were some-way to indicate to VOID_compare the field-order you wanted to use to compare on. There are a few ways I could think of doing this. 1) Use a thread-specific global variable (this doesn't recurse very easily). 2) Use the meta-object in the field specifier to indicate the order (the interface could still be something like .sort(order='f1') and a temporary data-type object is created and used). 3) Use a special key in the fields dictionary although this would require some fixes to all the code that cycles through the fields dictionary to recurse on structures. 4) Overload any of the other variables in the PyArray_Descr * structure. 5) Add a sort-order to the end of the PyArray_Descr * structure and a flag to the hasobject flag bits (that would be the last one available) that states that the Data-type object has the sort-order defined (so binary compatibilty is retained but the new feature can be used going forward). Any other ideas? -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Converting bool to float
Keith Goodman wrote: I had a hard time tracing a bug in my code. The culprit was this difference: x matrix([[True], [True], [True]], dtype=bool) 1.0 - x matrix([[ 0.], [ 0.], [ 0.]], dtype=float32) --- float32 1.0*x matrix([[ 1.], [ 1.], [ 1.]]) float64 numpy.__version__ '1.0rc1' Any chance that 1.0 - x could return dtype = float64? I'm surprised it doesn't. Both should follow bascially the same code-path. Perhaps there is a missing function loop or something. I'll look in to it. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Converting bool to float
Keith Goodman wrote: I had a hard time tracing a bug in my code. The culprit was this difference: x matrix([[True], [True], [True]], dtype=bool) 1.0 - x matrix([[ 0.], [ 0.], [ 0.]], dtype=float32) --- float32 1.0*x matrix([[ 1.], [ 1.], [ 1.]]) float64 numpy.__version__ '1.0rc1' Any chance that 1.0 - x could return dtype = float64? It looks like 1.0-x is doing the right thing. The problem is 1.0*x for matrices is going to float64. For arrays it returns float32 just like the 1.0-x This can't be changed at this point until 1.1 We will fix the bug in 1.0*x producing float64, however. I'm still not sure what's causing it, though. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Need more comments from scientific community on python-dev
I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information. There is a dearth of scientific-computing and number-crunching-aware people on python-dev. The result is that I sound like a lone voice arguing for something that nobody cares about. When, I don't think that is true. Please, please. If you want Python to grow support for the array interface (or something like it), then please speak up on python-dev. Even something as simple as I really see the need for a way to exchange data-format information between two objects sharing the buffer protocol can be helpful. You can post through the gmane newsgroup interface: gmane.comp.python.devel Find any of the posts on the PEP's I've introduced. Thanks for your help. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] dtype.hasobject value
Travis Oliphant wrote: Matthew Brett wrote: Hi, I notice that the value for: zeros((1,), dtype=object).dtype.hasobject is now 63, whereas previously it had been 1. Is this intended? Yes. We are using hasobject as an 8-bit flag now and separating out some of the concepts that make object arrays object arrays. This will be in 1.0.1 which will come out soon. It should be binary compatible with 1.0. Let me follow up with this. I've changed back dtype.hasobject so that it returns a Python boolean object. Thus comparisons to 1 should work fine. Under the covers it is (dtype-hasobject NPY_ITEM_HASOBJECT) == NPY_ITEM_HASOBJECT) There is a new attribute (flags) that contains all the flags in the hasobject variable. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Need more comments from scientific community on python-dev
Fernando Perez wrote: On 10/31/06, Rich Shepard [EMAIL PROTECTED] wrote: On Tue, 31 Oct 2006, Alan Isaac wrote: The easiest access to this discussion for me was http://news.gmane.org/group/gmane.comp.python.devel/ I cannot add to this discussion, but I REALLY hope others will help Travis out here. (A few have.) He is fielding a lot of questions, some of which look to me to be from individuals who are ready to have fairly strong opinions without really understanding the why of his proposals. All this is sufficiently far from my areas of expertise that I cannot contribute anything useful. Otherwise, I'd be happy to lend support. I actually worry about the same: I really would like to help, but after reading the whole discussion, I realized that the low-level details being asked and discussed are something I don't really know enough to say anything. And I don't want to sound simply saying 'Hey, Travis is great, listen to him!' to python-dev, since that (asides from looking silly) can be somewhat counter-productive. How does that sound, Travis? Is that something you think might help you, esp. since so many of us are feeling woefully underqualified to lend a useful hand in the actual discussion on python-dev? That would be great. I think a couple of things would also be useful. 1) Some way to indicate to python-dev that I'm actually speaking for more than just myself. So, while I agree that just supporting my PEP (which probably in reality needs work) without understanding it is counter-productive, a voice that says. We really do need this kind of functionality is at least one more voice. 2) Examples of sharing memory between two objects. PIL is the classic example and has some merit, but because the internal memory layout of the PIL is 'pointer-to-pointers' instead of 'big-chunk-of-memory' it's not a 1-1 match to NumPy and the array interface only can comunicate information about the mode. But, I can see other examples. PyMedia, PyGame, PyVideo? CVXOPT, PyVoxel. All of these seem to define their own objects which are basically just interpretations of chunks of memory. At one time, we might have said these should all be sub-classes of the ndarray. Now, we are thinking more along the lines of these should all expose an array interface. The array interface is still more bulky then it needs to be (it has to go through the attribute-lookup process which can be slow). It would be much better if the extended buffer protocol were available as a function-pointer on the type object of the type. If you have an application where you've ever wanted NumPy in the core. See if the extended buffer protocol serves your purposes and if you agree, voice your approval for the PEP. In my mind, the data-format PEP does not need to go through if there really is a better way to pass data-format information through the buffer protocol. But, the extended buffer protocol we *do* need. -Travis Regards, f ___ SciPy-user mailing list [EMAIL PROTECTED] http://projects.scipy.org/mailman/listinfo/scipy-user - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Need more comments from scientific community on python-dev
Sasha wrote: On 10/31/06, Travis Oliphant [EMAIL PROTECTED] wrote: I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information. I would love to help, but I feel that I will be on the other side of the disagreement. (That's why I reply here rather than on python-dev first.) Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter. Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format. What advantage do we gain by making every instance of a data-type object *also* a Python type object? We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument. I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change. I would, however, put effort into undertstanding ctypes objects as data-type objects. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Need more comments from scientific community on python-dev
Sasha wrote: On 10/31/06, Travis Oliphant [EMAIL PROTECTED] wrote: I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information. I would love to help, but I feel that I will be on the other side of the disagreement. (That's why I reply here rather than on python-dev first.) There's actually two issues here as well. 1) The extended buffer protocol 2) How to exchange data-format information through it. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Need more comments from scientific community on python-dev
Sasha wrote: On 10/31/06, Travis Oliphant [EMAIL PROTECTED] wrote: Yes, this is the key. I think we should better understand ctypes limitations before proposing an alternative. I already understand it's practical limitations --- type objects as data-type instances is too bulky and too restrictive. You have to be a Python type object in order to be a data-type object. I wish others would actually understand the difference. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Need more comments from scientific community on python-dev
Sasha wrote: On 10/31/06, Travis Oliphant [EMAIL PROTECTED] wrote: Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter. I read the whole python-dev thread before replying. I may be a little biased because I never liked somewhat cryptic letter codes in Numeric and the change of codes from Numeric to numpy did not contribute to my sympathy particularly when the convert script changed all unrelated instances of 'b' in my code to something else. The letter codes are not data-type objects. The letter codes are only there for historical reasons (and they are in the struct and array modules too so blame Python...) I am also not a big fan of record arrays. I believe that numeric data should be stored in inverted tables, where columns of homogeneous data are stored contiguously. Sure. I understand this. This was my argument too when Numarray was first proposed. But, how do you memory-map a record where the data is actually stored differently? Numeric's answer is you don't but that is un-acceptable. This is why record-arrays were created and the whole pandora's box of data-types was opened. Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format. Yes, but AFAIK this is a recent innovation. Numarray used type objects and Numeric simply used letter codes. Recent if 1 year is recent, and only if you worry precisely about when they were Python objects. Numeric always used a PyArray_Descr * structure (which was a PyObject_HEAD away from being a Python object) to describe data. The letter codes were just simple ways to represent those underlying structures (which could have been exposed to Python as data-types from the very beginning. What advantage do we gain by making every instance of a data-type object *also* a Python type object? I think the main advantage is that you can have instances: c_int(42) c_int(42) Of course, numpy has scalars for that, but ctypes also has fixed-length arrays, that are somewhat different from ndarrays: Sure, but you don't need an instance for every memory-layout description. If you want one, then great, ctypes gives it to you. But requiring data-type to be encoded in a type object is over-kill. We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument. But, isn't this someone else's head-ache? Someone has already gone through all these contortions, why not reuse the effort? No, it's not that simple. We have a headache whenever we want to do something like I just did and separate out the concepts of what makes a Python Object a Python object. Now, we don't just modify a simple C-structure (PyArray_Descr *), we have to modify a meta-type or a altered dictionary and get that change put in to ctypes. Inheriting from Python type objects is harder. People who have apparently never tried seem to think it's not but it is. Others on python-dev described some specific needs of ctypes that your datatype object does not address. Your point seems to be that numpy does not share these needs and could use a much simpler approach. No, that's not the point. The data-type object could easily be extended to accomodate those needs. The point is using data-types as instances of a regular Python object or data-types as instances of a type object. I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change. Can you present some use cases that illustrate the advantage of your datatype approach best? I will try to implement them with ctypes. This is missing the point. I have no doubt ctypes could be used on the Python side to describe a data-type. But, we need the PyArray_Descr * structure in NumPy. What are you going to replace the PyArray_Descr * structure with? That is the point. I would, however, put effort into undertstanding ctypes objects as data-type objects. Yes, this is the key. I think we should better understand ctypes limitations before proposing an alternative. Please understand what I meant. I meant putting effort into getting PyArray_DescrConverter to allow ctypes inputs and convert to the appropriate PyArray_Descr * structure. I already understand ctypes objects. I want the dtype() command to also understand them. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1
Re: [Numpy-discussion] glibc invalid pointer error
Fernando Perez wrote: On 10/30/06, David Huard [EMAIL PROTECTED] wrote: Hi, I have a script that crashes, but only if it runs over 9~10 hours, with the following backtrace from gdb. The script uses PyMC, and repeatedly calls ( 100) likelihood functions written in fortran and wrapped with f2py. Numpy: 1.0.dev3327 Python: 2.4.3 This sounds awfully reminiscent of the bug I recently mentioned: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3312099 It actually looks very much like it. I think the problem may be in f2py or in one of the C-API calls where-in there is a reference-count problem with the built-in data-type objects. NumPy won't try to free those anymore which will solve the immediate problem, but there is still a reference-count problem somewhere. The reference to the data-type objects is consumed by constructors that take PyArray_Descr * arguments. So, you often need to INCREF before passing to those constructors. It looks like this INCREF is forgotten in some extension module (perhaps in f2py or PyMC). It's possible it's in NumPy itself, though I've re-checked the code lots of times looking for that specific problem. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] glibc invalid pointer error
Fernando Perez wrote: On 10/30/06, David Huard [EMAIL PROTECTED] wrote: Hi, I have a script that crashes, but only if it runs over 9~10 hours, with the following backtrace from gdb. The script uses PyMC, and repeatedly calls ( 100) likelihood functions written in fortran and wrapped with f2py. Numpy: 1.0.dev3327 Python: 2.4.3 This sounds awfully reminiscent of the bug I recently mentioned: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3312099 We left a fresh run over the weekend, but my office mate is currently out of the office and his terminal is locked, so I don't know what the result is. I'll report shortly: we followed Travis' instructions and ran with a fresh SVN build which includes the extra warnings he added to the dealloc routines. You may want to try the same advice, perhaps with information from both of us the gurus may zero in on the problem, if indeed it is the same. I talked about the reference counting issue. One problem is not incrementing the reference count when it needs to be. The other problem could occur if the reference-count was not decremented when it needed to be and the reference count wrapped from MAX_LONG to 0. This could also create the problem and would be expected for long-running processes. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] glibc invalid pointer error
David Huard wrote: Ok, I'll update numpy and give it another try tonight. I just fixed some reference-count problems in f2py today. These were of the variety that there was a missing decref that would cause the reference count of certain often-used data-types to increase without bound and eventually wrap (to 0) in long-running processes using f2py. I suspect this is the fundamental problem in both cases. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Strange numpy.argmax behavior on object arrays in numpy 1.0.
Tom Denniston wrote: I recently upgraded to numpy 1.0 from 1.0b5. I noticed that numpy.argmax behavior is very strange on object arrays. See below: (Pdb) numpy.__version__ '1.0' (Pdb) numpy.argmax(numpy.array([2, 3], dtype=object)) 0 (Pdb) numpy.argmax(numpy.array([2, 3], dtype=int)) 1 (Pdb) numpy.argmax(numpy.array([2, 3], dtype=object), axis=0) 0 I would expect the argmax to behave the same on the dtype=int and dtype=object examples but it doesn't. Am I missing some subtelty or is this just a bug? 1.0 is the most recent version, right? This is a bug. I've fixed in in SVN. Thanks for the test. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] recarray.__setattr__ bug?
Michael McNeil Forbes wrote: In article [EMAIL PROTECTED], Travis Oliphant [EMAIL PROTECTED] wrote: Hmm I know that the code was changed at some point a few months ago specifically to this behavior because of some concerns Perry, Chris (people at STScI) had. Originally, field names came first, but we changed it so they could set known attributes of a record array even if they were also field names. This may be an unintentional side-effect.So, let's not just change things again and create problems for them. Were any test cases generated for this? Changing the code did not break anything: if this is important, a test should probably be added. You are right. In fact there are lots of tests that need to be added. Stefan as been doing a great job of regression testing. I've committed some fixes to this particular issue that allows setattr to be used to set field names. Now 1) The standard setattr is tried first 2) If it fails, then the field-dictionary is checked to see if there is a field with that name. a) If not, then the error created by 1 is raised b) if the name is a field, then an attempt is made to set the field with that name 3) If the standard setattr succeeds then either i) a built-in attribute was correctly set or ii) a new attribute was created and attached to the dictionary. a) If the attribute is not in the fields, then return b) Otherwise, see if this name was in the .__dict__ before the call (if it's not then we just added it so delete it). i) If the deletion succeeds then set set the field ii If the deletion fails, then return the result of standard setattr setting. I've added a test for some of these behaviors. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] recarray.__setattr__ bug?
Michael McNeil Forbes wrote:
Is the following the desired behaviour for setting recarray attributes?
This seems to clash with the semantics for arrays.
from numpy import *
a = array([1,2,3])
b = a.view([('x',int),('y',int),('z',int)])
r = b.view(recarray)
b['x'] = 0
r.y = 0
a
array([0, 2, 3])
r.x
array([0])
r.y
0
Setting r.y creates a local variable in r.__dict__ which is then
accessed rather than the array element referred to by fielddict.
Hmm I know that the code was changed at some point a few months ago
specifically to this behavior because of some concerns Perry, Chris
(people at STScI) had. Originally, field names came first, but we
changed it so they could set known attributes of a record array even if
they were also field names.
This may be an unintentional side-effect.So, let's not just change
things again and create problems for them.
-Travis
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] A recarray of arrays
Pierre GM wrote:
Folks,
What is the easiest way to define a recarray of arrays ?
For example, I'd need something like that:
Given three arrays
x = N.arange(5)
y = x+1
z = N.sqrt(x)
and a list of names:
n = ['x','y','z']
Define a 3-record array with two fields: the first one being a ndarray (in x,
y or), the second a string (in n).
I'm a bit at loss with the definition of the corresponding dtype...
I'm a little unclear one what you want.
Here is a data-type whose first field is a name and whose second field
is a 5-element array:
dt = N.dtype([('name', 'S10'), ('data', float, (5,))])
Then:
a = array([('x',x),('y',y),('z',z)],dt)
gives
array([('x', [0.0, 1.0, 2.0, 3.0, 4.0]), ('y', [1.0, 2.0, 3.0, 4.0, 5.0]),
('z', [0.0, 1.0, 1.4142135623730951, 1.7320508075688772, 2.0])],
dtype=[('name', '|S10'), ('data', 'f8', (5,))])
-Travis
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Defining custom types
Jonathan Wang wrote: On 10/26/06, *Travis Oliphant* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Okay, is my understanding here correct? I am defining two type descriptors: PyArray_Descr mxNumpyType - describes the Numpy array type. PyTypeObject mxNumpyDataType - describes the data type of the contents of the array (i.e. mxNumpyType-typeobj points to this), inherits from PyDoubleArrType_Type and overrides some fields as mentioned above. The nomenclature is that mxNumPyType is the data-type of the array and your PyTypeObject is the type of the elements of the array. So, you have the names a bit backward. So, to correspond with the way I use the words type and data-type, I would name them: PyArray_Descr mxNumpyDataType PyTypeObject mxNumpyType Okay, I will use this convention going forwards. And the getitem and setitem functions are designed to only give/take PyObject* of type mxDateTime. These are in the 'f' member of the PyArray_Descr structure, so presumably you have also filled in your PyArray_Descr structure with items from PyArray_DOUBLE? That's correct. I have all members of the 'f' member identical to that from PyArray_DOUBLE, except: mxNumpyType-f-dotfunc = NULL; mxNumpyType-f-getitem = date_getitem; mxNumpyType-f-setitem = date_setitem; mxNumpyType-f-cast[PyArray_DOUBLE] = (PyArray_VectorUnaryFunc*) dateToDouble; mxNumpyType-f-cast[PyArray_OBJECT] = (PyArray_VectorUnaryFunc*) dateToObject; All other cast functions are NULL. If I redefine the string function, I encounter another, perhaps more serious problem leading to a segfault. I've defined my string function to be extremely simple: def printer(arr): ... return str(arr[0]) Now, if I try to print an element of the array: mxArr[0] I get to this stack trace: #0 scalar_value (scalar=0x814be10, descr=0x5079e0) at scalartypes.inc.src:68 #1 0x0079936a in PyArray_Scalar (data=0x814cf98, descr=0x5079e0, base=0x814e7a8) at arrayobject.c:1419 #2 0x007d259f in array_subscript_nice (self=0x814e7a8, op=0x804eb8c) at arrayobject.c:1985 #3 0x00d17dde in PyObject_GetItem (o=0x814e7a8, key=0x804eb8c) at Objects/abstract.c:94 (Note: for some reason gdb claims that arrayobject.c:1985 is array_subscript_nice, but looking at my source this line is actually in array_item_nice. *boggle*) But scalar_value returns NULL for all non-native types. So, destptr in PyArray_Scalar is set to NULL, and the call the copyswap segfaults. Perhaps scalar_value should be checking the scalarkind field of PyArray_Descr, or using the elsize and alignment fields to figure out the pointer to return if scalarkind isn't set? Hmmm... It looks like the modifications to scalar_value did not take into account user-defined types. I've added a correction so that user-defined types will use setitem to set the scalar value into the array. Presumably your setitem function can handle setting the array with scalars of your new type? I've checked the changes into SVN. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] dtype() examples that used to work
Joris De Ridder wrote:
Hi,
The following did once work in NumPy:
dtype([int16])
dtype([[int16]])
dtype([uint,int32])
dtype(['f8','S10'])
but now they all generate a TypeError: data type not understood. Why?
I could not find the answer in help(dtype), the RecordArrays tutorial on
scipy.org, the NumPy release notes, nor in the NumPy book.
It's in the release notes. Look in the section from 1.0b2 to 1.0b3.
It's highlighted with a Note:
*NOTE: * When creating data-type objects from lists, the list must be in
array descriptor format. The list of data-type objects approach is no
longer supported due to its ambiguity with the very common array
descriptor approach. You can still get this behavior using a
comma-separated string or changing your list to an array_descriptor:
[('', x) for x in oldlist]
-Travis
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] pyaudio, a module to make noise from numpy arrays
David Cournapeau wrote: Hi, I announce the first release of pyaudio, a module to make noise from numpy arrays (read, write and play audio files with numpy arrays). Very nice. Thank you. I'd like to see exactly this kind of thing for video files too. We can get a lot of mileage out of ctypes. By the way. I noticed your setup.py file is missing a .todict() to convert the configuration object into a dictionary that setup(** ) can handle. if __name__ == __main__: from numpy.distutils.core import setup setup(**configuration(top_path='').todict()) You might also check to see if libsndfile can be imported during the setup phase and then warn the user before installing that they need it. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Defining custom types
Jonathan Wang wrote: On 10/27/06, *Travis Oliphant* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: If I redefine the string function, I encounter another, perhaps more serious problem leading to a segfault. I've defined my string function to be extremely simple: def printer(arr): ... return str(arr[0]) Now, if I try to print an element of the array: mxArr[0] I get to this stack trace: #0 scalar_value (scalar=0x814be10, descr=0x5079e0) at scalartypes.inc.src:68 #1 0x0079936a in PyArray_Scalar (data=0x814cf98, descr=0x5079e0, base=0x814e7a8) at arrayobject.c:1419 #2 0x007d259f in array_subscript_nice (self=0x814e7a8, op=0x804eb8c) at arrayobject.c:1985 #3 0x00d17dde in PyObject_GetItem (o=0x814e7a8, key=0x804eb8c) at Objects/abstract.c:94 (Note: for some reason gdb claims that arrayobject.c:1985 is array_subscript_nice, but looking at my source this line is actually in array_item_nice. *boggle*) But scalar_value returns NULL for all non-native types. So, destptr in PyArray_Scalar is set to NULL, and the call the copyswap segfaults. Perhaps scalar_value should be checking the scalarkind field of PyArray_Descr, or using the elsize and alignment fields to figure out the pointer to return if scalarkind isn't set? Hmmm... It looks like the modifications to scalar_value did not take into account user-defined types. I've added a correction so that user-defined types will use setitem to set the scalar value into the array. Presumably your setitem function can handle setting the array with scalars of your new type? I've checked the changes into SVN. Do there also need to be changes in scalartypes.inc.src to use getitem if a user-defined type does not inherit from a Numpy scalar? This needs to be clarified. I don't think it's possible to do it without inheriting from a numpy scalar at this point (the void numpy scalar can be inherited from and is pretty generic). I know I was not considering that case when I wrote the code. i.e. at scalartypes.inc.src:114 we should return some pointer calculated from the PyArray_Descr's elsize and alignment field to get the destination for the custom scalar type to be copied. I think this is a good idea. I doubt it's enough to fix all places that don't inherit from numpy scalars, but it's a start. It seems like we need to figure out where the beginning of the data is for the type which is assumed to be defined on alignment boundaries after a PyObject_HEAD (right)? This could actually be used for everything and all the switch and if statements eliminated. I think the alignment field is the only thing needed, though. I don't see how I would use the elsize field? As it stands, if the user-defined type does not inherit from a Numpy scalar, lots of things continue to break. Not surprising, I did not make sure and support this. Furthermore it seems like the scalar conversions prefer the builtin types, but it seems to me that the user-defined type should be preferred. I'm not sure what this means. i.e. if I try to get an element from my mxDateTime array, I get a float back: mxArr[0] = DateTime.now() mxArr[0][0] 732610.60691268521 Why can you index mxArr[0]? What is mxArr[0]? If it's a scalar, then why can you index it? What is type(mxArr[0])? But what I really want is the mxDateTime, which, oddly enough, is what happens if I use tolist(): mxArr.tolist()[0] [DateTime object for '2006-10-27 14:33:57.25' at a73c60] That's not surprising because tolist just calls getitem on each element in the array to construct the list. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Unit test error with numpy rc3
Ted Horst wrote: On Mac OS X tiger (10.4) ppc, long double has increased precision but the same range as double (it really is 128 bits not 80, btw), so e**1000 is inf, so this is not really an error. Thanks for the clarification. Long-double is not standard across platforms with different platforms choosing to do different things with the extra bytes.This helps explain one more platform. I'm not sure what is the right thing to do in the test, check for overflow? Also, finfo has never worked properly for this type. In machar.py is the code that runs to detect all of the parameters. I think the code should be moved to C as detecting precision on a long-double takes too long. The overflow check is a good idea. The test should probably check for overflow and not try to run if it's detected. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Defining custom types
Jonathan Wang wrote: I'm trying to write a Numpy extension that will encapsulate mxDateTime as a native Numpy type. I've decided to use a type inherited from Numpy's scalar double. However, I'm running into all sorts of problems. I'm using numpy 1.0b5; I realize this is somewhat out of date. Cool. The ability to create your own data-types (and define ufuncs for them) is a feature that I'd like to see explored. But, it has not received a lot of attention and so you may find bugs along the way. We'll try to fix them quickly as they arise (and there will be bug fix releases for 1.0). But, what do you mean inheriting from NumPy's double for your scalar data-type. This has significant implications. To define a new data-type object (that doesn't build from the VOID data-type), you need to flesh out the PyArray_Descr * structure and this can only be done in C. Perhaps you are borrowing most entries in the structure builtin double type and then filling in a few differently like setitem and getitem? Is that accurate? For all the examples below, assume that I've created a 1x1 array, mxArr, with my custom type. The interface used by Array_FromPyScalar does not conform with the documentation's claim that a negative return value indicates an error. You must be talking about a different function. Array_FromPyScalar is an internal function and not a C-API call. It also returns a PyObject * not an integer. So, which function are you actually referring to? The return code from setitem is not checked. Instead, the code depends on a Python error being set. This may be true, but how is it a problem? I seem to be able to load values into the array, but I can't extract anything out of the array, even to print it. In gdb I've verified that loading DateTime.now() correctly puts a float representation of the date into my array. However, if I try to get the value out, I get an error: mxArr[0] = DateTime.now() mxArr[0] Traceback (most recent call last): File stdin, line 1, in ? File /usr/lib/python2.4/site-packages/numpy/core/numeric.py, line 391, in array_repr ', ', array() File /usr/lib/python2.4/site-packages/numpy/core/arrayprint.py, line 204, in array2string separator, prefix) File /usr/lib/python2.4/site-packages/numpy/core/arrayprint.py, line 160, in _array2string format = _floatFormat(data, precision, suppress_small) File /usr/lib/python2.4/site-packages/numpy/core/arrayprint.py, line 281, in _floatFormat non_zero = _uf.absolute(data.compress(_uf.not_equal(data, 0))) TypeError: bad operand type for abs() I'm not sure why it's trying to call abs() on my object to print it. Because that's the implication of inheriting from a double. It's just part of the code that tries to format your values into an array (notice the _floatFormat). I actually borrowed this code from numarray so I can't speak to exactly what it's doing without more study. I have a separate PyNumberMethods attached to my object type, copied from the float scalar type, and nb_absolute is set to 0. When I break at the various functions I've registered, the last thing Numpy tries to do is cast my custom data type to an object type (which it does so successfully) via _broadcast_cast. Don't confuse the Python object associated when an element of the array is extracted and the data-type of the array. Also don't confuse the PyNumberMethods of the scalar object with the ufuncs. Defining PyNumberMethods won't usually give you the ability to calculate ufuncs. Perhaps you just want to construct an object array of mxDateTime's. What is the reason you want to define an mxDateTime data-type? -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy-1.0 regress failure on OpenBSD
Damien Miller wrote: Hi, I have just got around to updating OpenBSD's numpy port from 1.0b1 to 1.0 and am running into the following regress failure: .Warning: overflow encountered in exp F... == FAIL: Ticket #112 -- Traceback (most recent call last): File /usr/ports/math/py-numpy/w-py-numpy-1.0/fake-i386/usr/local/lib/python2.4/site-packages/numpy/core/tests/test_regression.py, line 220, in check_longfloat_repr assert(str(a)[1:9] == str(a[0])[:8]) AssertionError -- Ran 513 tests in 3.471s FAILED (failures=1) The variable 'a' seems to contain '[Inf]', so the failure appears related to the overflow warning. Any clues on how I can debug this further? Unless you want to help with tracking how long double is interpreted on several platforms, then just ignore the test. (It actually wasn't being run in 1.0b1). -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Defining custom types
It's just confusing as the documentation indicates that the setitem
function should return 0 for success and a negative number for
failure. But within Array_FromPyScalar, we have:
ret-descr-f-setitem(op, ret-data, ret);
if (PyErr_Occurred()) {
Py_DECREF(ret);
return NULL;
} else {
return (PyObject *)ret;
}
I see the problem. We are assuming an error is set on failure, so both
-1 should be returned and an error condition set for your own setitem
function.This is typical Python behavior. I'll fix the documentation.
I seem to be able to load values into the array, but I can't extract
anything out of the array, even to print it. In gdb I've
verified that
loading DateTime.now() correctly puts a float representation of the
date into my array. However, if I try to get the value out, I get an
error:
mxArr[0] = DateTime.now()
mxArr[0]
Traceback (most recent call last):
File stdin, line 1, in ?
File /usr/lib/python2.4/site-packages/numpy/core/numeric.py,
line
391, in array_repr
', ', array()
File /usr/lib/python2.4/site-packages/numpy/core/arrayprint.py,
line 204, in array2string
separator, prefix)
File /usr/lib/python2.4/site-packages/numpy/core/arrayprint.py,
line 160, in _array2string
format = _floatFormat(data, precision, suppress_small)
File /usr/lib/python2.4/site-packages/numpy/core/arrayprint.py,
line 281, in _floatFormat
non_zero = _uf.absolute(data.compress(_uf.not_equal(data, 0)))
TypeError: bad operand type for abs()
I'm not sure why it's trying to call abs() on my object to print it.
Because that's the implication of inheriting from a double. It's
just
part of the code that tries to format your values into an array
(notice
the _floatFormat). I actually borrowed this code from numarray so I
can't speak to exactly what it's doing without more study.
Hmm, so does Numpy ignore the tp_repr and tp_str fields in the
PyTypeObject of the underlying type? admittedly haven't had a chance
to look at this code closely yet.
How arrays print is actually user-settable. The default printing
function does indeed ignore tp_repr and tp_str of the underlying scalar
objects in order to be able to set precisions.
Now, we could probably fix the default printing function to actually use
the tp_repr and/or tp_str fields of the corresponding scalar objects.
This is worth filing a ticket about.
In the mean time you can create a new array print function that checks
for your data-type as the type of the array and then does something
different otherwise it calls the old function. Then, register this
function as the print function for arrays.
I have a separate PyNumberMethods attached to my object type, copied
from the float scalar type, and nb_absolute is set to 0. When I
break
at the various functions I've registered, the last thing Numpy
tries
to do is cast my custom data type to an object type (which it
does so
successfully) via _broadcast_cast.
Don't confuse the Python object associated when an element of the
array
is extracted and the data-type of the array. Also don't confuse the
PyNumberMethods of the scalar object with the ufuncs. Defining
PyNumberMethods won't usually give you the ability to calculate
ufuncs.
Okay, is my understanding here correct? I am defining two type
descriptors:
PyArray_Descr mxNumpyType - describes the Numpy array type.
PyTypeObject mxNumpyDataType - describes the data type of the contents
of the array (i.e. mxNumpyType-typeobj points to this), inherits from
PyDoubleArrType_Type and overrides some fields as mentioned above.
The nomenclature is that mxNumPyType is the data-type of the array and
your PyTypeObject is the type of the elements of the array. So, you
have the names a bit backward.
So, to correspond with the way I use the words type and data-type, I
would name them:
PyArray_Descr mxNumpyDataType
PyTypeObject mxNumpyType
And the getitem and setitem functions are designed to only give/take
PyObject* of type mxDateTime.
These are in the 'f' member of the PyArray_Descr structure, so
presumably you have also filled in your PyArray_Descr structure with
items from PyArray_DOUBLE?
I guess it's not clear to me whether the abs() referred to by the
error is an abs() ufunc or the nb_absolute pointer in the
PyNumberMethods. Let me try overriding ufuncs and get back to you...
Perhaps you just want to construct an object array of mxDateTime's.
What is the reason you want to define an mxDateTime data-type?
Currently I am using an object array of mxDateTime's, but it's rather
frustrating that I can't treat them as normal floats
Re: [Numpy-discussion] Strange results when sorting array with fields
Albert Strasheim wrote: Hello all I'm trying to sort an array with two fields, but I'm getting a result that doesn't seem to make sense. What I tried (first attempt): I have two 2-D arrays. I would like to sort one based on the sort of the other. I managed to do this with argsort. However, the fancy indexing required to get the sorted array using what argsort returned was very slow. I followed this example: http://www.scipy.org/Numpy_Example_List#head-9f8656795227e3c43e849c6c0435eee b32afd722 What I tried (second attempt): I created an array with two fields. I think/hope/expected that sorting the array would sort it on the first field in the dtype and then on the second. This is *much* faster than the fancy indexing approach. That kind of sorting is what lexsort does (although with indexes) and is more complicated than what the sort routine does in NumPy. The sorting routines in NumPy use the output of the comparison operator for the type to compute the result. An array with fields is of type void. Right now, the VOID_compare routine is equivalent to the STRING_compare routine (i.e. raw bytes are compared). I doubt this will do what you want in most cases. It would be possible to adapt this compare routine when fields are present and so something like compare the first field first and the second field only if the first is equal. But, this would require a bit of work and is probably best left for 1.0.1 or later. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Strange results when sorting array with fields
Albert Strasheim wrote: Hello all I'm trying to sort an array with two fields, but I'm getting a result that doesn't seem to make sense. What I tried (first attempt): I have two 2-D arrays. I would like to sort one based on the sort of the other. I managed to do this with argsort. However, the fancy indexing required to get the sorted array using what argsort returned was very slow. I followed this example: http://www.scipy.org/Numpy_Example_List#head-9f8656795227e3c43e849c6c0435eee b32afd722 What I tried (second attempt): I created an array with two fields. I think/hope/expected that sorting the array would sort it on the first field in the dtype and then on the second. This is *much* faster than the fancy indexing approach. O.K. I lied, I realized that my comments in the VOID_compare code were silly (about being unable to define or ). It makes sense to just define them based on the first field, then the second field (if the first field is equal), and so forth. Obviously this is not the only way one could define sorting (any field could be used as the first field, and so forth. But, is is a fairly obvious default to use the first field. It turns out, it was not so difficult to implement this and is now in SVN. So, the VOID_compare now does something a little more intelligent when fields are defined which means that record arrays can now be lexicographically sorted more easily than using lexsort and take (as long as the fields are ordered according to how you want the sort to proceed). -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Strange and hard to reproduce crash
Fernando Perez wrote: Hi all, two colleagues have been seeing occasional crashes from very long-running code which uses numpy. We've now gotten a backtrace from one such crash, unfortunately it uses a build from a few days ago: This looks like a reference-count problem on the data-type objects (probably one of the builtin ones is trying to be released). The reference count problem is probably hard to track down. A quick fix is to not allow the built-ins to be freed (the attempt should never be made, but if it is, then we should just incref the reference count and continue rather than die). Ideally, the reference count problem should be found, but other-wise I'll just insert some print statements if the attempt is made, but not actually do it as a safety measure. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Strange and hard to reproduce crash
Fernando Perez wrote:
On 10/23/06, Travis Oliphant [EMAIL PROTECTED] wrote:
Fernando Perez wrote:
Hi all,
two colleagues have been seeing occasional crashes from very
long-running code which uses numpy. We've now gotten a backtrace from
one such crash, unfortunately it uses a build from a few days ago:
This looks like a reference-count problem on the data-type objects
(probably one of the builtin ones is trying to be released). The
reference count problem is probably hard to track down.
A quick fix is to not allow the built-ins to be freed (the attempt
should never be made, but if it is, then we should just incref the
reference count and continue rather than die).
Ideally, the reference count problem should be found, but other-wise
I'll just insert some print statements if the attempt is made, but not
actually do it as a safety measure.
If you point me to the right place in the sources, I'll be happy to
add something to my local copy, rebuild numpy and rerun with these
print statements in place.
I've placed them in SVN (r3384):
arraydescr_dealloc needs to do something like.
if (self-fields == Py_None) {
print something
incref(self)
return;
}
Most likely there is a missing Py_INCREF() before some call that uses
the data-type object (and consumes it's reference count) --- do you have
any Pyrex code (it's harder to get it right with Pyrex).
I realize this is probably a very difficult problem to track down, but
it really sucks to run a code for 4 days only to have it explode at
the end. Right now this is starting to be a serious problem for us as
we move our codes into large production runs, so I'm willing to put in
the necessary effort to track it down, though I'll need some guidance
from our gurus.
Tracking the reference count of the built-in data-type objects should
not be too difficult. First, figure out which one is causing problems
(if you still have the gdb traceback, then go up to the
arraydescr_dealloc function and look at self-type_num and self-type).
Then, put print statements throughout your code for the reference count
of this data-type object.
Something like,
sys.getrefcount(numpy.dtype('float'))
would be enough at a looping point in your code.
Good luck,
-Travis
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Release of 1.0 coming
Albert Strasheim wrote: Hey Travis -Original Message- From: [EMAIL PROTECTED] [mailto:numpy- [EMAIL PROTECTED] On Behalf Of Travis Oliphant Sent: Tuesday, October 24, 2006 12:32 AM To: Discussion of Numerical Python Subject: [Numpy-discussion] Release of 1.0 coming The long awaited day is coming--- Wednesday is the target. Please submit problems before Tuesday (tomorrow). Nothing but bug-fixes are being changed right now. Some Valgrind warnings that you might want to look at: http://projects.scipy.org/scipy/numpy/ticket/360 fixed. Maybe faltet could provide some code to reproduce this problem: http://projects.scipy.org/scipy/numpy/ticket/355 Looked at it and couldn't see what could be wrong. Need code to reproduce the problem. I think this ndpointer issue has been resolved (Stefan?): http://projects.scipy.org/scipy/numpy/ticket/340 Yes it has. Fixed. I think ctypes 1.0.1 is required for ndpointer to work, so we might consider some kind of version check + warning on import? Not sure about that. It worked for me using ctypes 1.0.0. Maybe a Python at-exit handler can be used to avoid the add_docstring leaks described here: http://projects.scipy.org/scipy/numpy/ticket/195 I'm not too concerned about this. Whether we release the memory right before exiting or just let the O.S. do it when the process quits seems rather immaterial. It would be a bit of work to implement so the cost / benefit ratio seems way to high. Also, what's the story with f2py? It seems Pearu is still making quite a few changes in the trunk as part of F2PY G3. Pearu told me not to hold up NumPy 1.0 because f2py g3 is still a ways away. His changes should not impact normal usage of f2py. I suspect NumPy 1.0.1 will contain f2py g3 -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] slicing suggestion
JJ wrote: Hello. I have a suggestion that might make slicing using matrices more user-friendly. I often have a matrix of row or column numbers that I wish to use as a slice. If K was a matrix of row numbers (nx1) and M was a nxm matrix, then I would use ans = M[K.A.ravel(),:] to I had thought that something like that would be useful. It's called .A1 -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem introduced after 1.0rc2 on AIX with xlc
Brian Granger wrote: Hi, i am running numpy on aix compiling with xlc. Revision 1.0rc2 works fine and passes all tests. But 1.0rc3 and more recent give the following on import: Most likely the error-detection code is not working on your platform. The platform dependent stuff is not that difficult. I tried to implement something for AIX, but very likely got it wrong (and don't have a platform to test it on). It is the UFUNC_CHECK_STATUS that must be implemented. Perhaps, we can do a simple check and disable the error modes: seterr(all='ignore') will work and turn-off error-detection on your platform. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] adding an attribute to an nd-array
Stefan van der Walt wrote:
On Wed, Oct 18, 2006 at 09:17:49PM -0400, Pierre GM wrote:
On Wednesday 18 October 2006 20:29, Stefan van der Walt wrote:
A quick question on extending numpy arrays: is it possible to easily
add an attribute to an ndarray?
It might be easier to create a subclass: pleasehave a look here:
http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/test_subclasses.py
That's a tiny example of subclassing ndarrays, with some extra attributes.
(BTW, I'm not sure that's the most obvious place where to look: if it turns
out to be useful, I'll put it on the scipy wiki)
Thanks very much, Pierre.
If I understand correctly, the following should work:
import numpy as N
class InfoArray(N.ndarray):
def __new__(info_arr_cls,arr,info={}):
info_arr_cls.info = info
return N.array(arr).view(info_arr_cls)
When does __array_finalize__ get called, and is it always necessary to
specify it?
It gets called whenever a new array is created. No, it is not necessary
to specify it.
-Travis
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Product of an empty sequence
Ivan Vilata i Balaguer wrote: Today I was surprised by this:: import numpy numpy.__version__ '1.0.dev3341' numpy.prod(()) 1.0 Wouldn't be 0.0 a more natural answer? I understand that 1 is neutral on product operations, but I still can't see why 1.0 is a better answer. It's the identity for the multiplication ufunc. Sorry for the clueless question, but I have searched lists, docstrings and web and found nothing about this. (BTW, the definitions of ``prod()`` and ``product()`` are still identical as noted by Sebastian Haase some time ago.) Is this a problem? -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Unit test error with numpy rc3
Tim Hochberg wrote: Travis Oliphant wrote: Tim Hochberg wrote: Rudolph van der Merwe wrote: I get the following error with RC3 on a RHE Linux box: Python 2.4.3 (#4, Mar 31 2006, 12:12:43) [GCC 3.4.5 20051201 (Red Hat 3.4.5-2)] on linux2 Type help, copyright, credits or license for more information. import numpy numpy.__version__ '1.0rc3' On a visual studio build, I'm getting a real failure though: FAIL: Ticket #112 -- Traceback (most recent call last): File C:\Python24\lib\site-packages\numpy\core\tests\test_regression.py, line 219, in check_longfloat_repr assert(str(a)[1:9] == str(a[0])[:8]) AssertionError The code in question is dependent on the spelling of INF(or at least the *length* of the spelling) on a given platform which is why it's failing. Actually, you shouldn't be getting an INF at all.This is what the test is designed to test for (so I guess it's working). The test was actually written wrong and was never failing because previously keyword arguments to ufuncs were ignored. Can you show us what 'a' is on your platform. Sure: import numpy as N a = N.exp(N.array([1000],dtype=N.longfloat)) Warning: overflow encountered in exp a array([1.#INF], dtype=float64) O.K. We need to modify the test in case to check and see that the size of longfloat isn't the same as double. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] adding an attribute to an nd-array
Stefan van der Walt wrote:
On Wed, Oct 18, 2006 at 09:17:49PM -0400, Pierre GM wrote:
On Wednesday 18 October 2006 20:29, Stefan van der Walt wrote:
A quick question on extending numpy arrays: is it possible to easily
add an attribute to an ndarray?
It might be easier to create a subclass: pleasehave a look here:
http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/test_subclasses.py
That's a tiny example of subclassing ndarrays, with some extra attributes.
(BTW, I'm not sure that's the most obvious place where to look: if it turns
out to be useful, I'll put it on the scipy wiki)
Thanks very much, Pierre.
If I understand correctly, the following should work:
import numpy as N
class InfoArray(N.ndarray):
def __new__(info_arr_cls,arr,info={}):
info_arr_cls.info = info
return N.array(arr).view(info_arr_cls)
When does __array_finalize__ get called, and is it always necessary to
specify it?
Actually something as simple as
class InfoArray(N.ndarray):
pass
will allow you to add attributes to InfoArray.
I just learned about how to allow built-ins to have attributes assigned
to their instances. It's actually pretty easy because of Python
support for it --- but it comes at a cost. You have to add a dictionary
to the PyArrayObject structure, create that dictionary when the ndarray
is allocated, and set the tp_dictoffset in the TypeObject structure to
its location in PyArrayObject. It takes 4 lines of code with the cost of
creating a new dictionary for every ndarray.
I don't think the extra bytes for every ndarray object are worth it,
given how easy it is to sub-class and create your own ndarray that can
have attributes attached.What are others opinions.
-Travis
P.S. Here is the patch that adds it:
Index: numpy/core/include/numpy/ndarrayobject.h
===
--- numpy/core/include/numpy/ndarrayobject.h(revision 3366)
+++ numpy/core/include/numpy/ndarrayobject.h(working copy)
@@ -1172,6 +1172,7 @@
PyArray_Descr *descr; /* Pointer to type structure */
int flags; /* Flags describing array -- see below*/
PyObject *weakreflist; /* For weakreferences */
+PyObject *instancedict; /* For instance attributes */
} PyArrayObject;
#define NPY_AO PyArrayObject
Index: numpy/core/src/arrayobject.c
===
--- numpy/core/src/arrayobject.c(revision 3366)
+++ numpy/core/src/arrayobject.c(working copy)
@@ -1906,6 +1906,8 @@
if (self-weakreflist != NULL)
PyObject_ClearWeakRefs((PyObject *)self);
+Py_DECREF(self-instancedict);
+
if(self-base) {
/* UPDATEIFCOPY means that base points to an
array that should be updated with the contents
@@ -5305,6 +5307,7 @@
self-descr = descr;
self-base = (PyObject *)NULL;
self-weakreflist = (PyObject *)NULL;
+self-instancedict = PyDict_New();
if (nd 0) {
self-dimensions = PyDimMem_NEW(2*nd);
@@ -6689,7 +6692,7 @@
0,/* tp_dict */
0,/* tp_descr_get */
0,/* tp_descr_set */
-0,/* tp_dictoffset */
+offsetof(PyArrayObject, instancedict),/* tp_dictoffset */
(initproc)0, /* tp_init */
array_alloc, /* tp_alloc */
(newfunc)array_new, /* tp_new */
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] adding an attribute to an nd-array
Christopher Barker wrote: Travis Oliphant wrote: Actually something as simple as class InfoArray(N.ndarray): pass will allow you to add attributes to InfoArray. Well, sure, but how the heck do you initialize it? You use the same constructor as ndarray has. numpy.info(numpy.ndarray) If you want an array-like function that produces the array, you do array(obj).view(InfoArray) or wrap that up in your own function. There are many over-lapping ways to construct ndarray's. You can use all of these ways to construct instances of your own type by getting the ndarray and using the .view() method. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] The NumPy Fortran-ordering quiz
Stefan van der Walt wrote: One last case, which confuses me still (probably because it is 04:16am): In [41]: x = N.array([[0,1,2],[3,4,5]],order='F') In [42]: x Out[42]: array([[0, 1, 2], [3, 4, 5]]) I assume the data is now stored in memory as [0 3 1 4 2 5] (column-wise) If I now do x.reshape((3,2),order='C') i.e. take that block of memory, assume it is in 'C' order, and make its shape (3,2), I expect [[0 3] [1 4] [2 5]] but get [[1 2] [3 4] [5 6]] I'm obviously missing something trivial -- I'll try again tomorrow. I think I see what is going on and where people are getting tripped up. You have to remember, that to NumPy it doesn't semantically matter what the ordering of the array is. There is no guarantee that C- order or Fortran-order is *ever* preserved through an operation. Because, in fact the general memory model of the array has no defined order. It's defined by the strides array. It just so happens that two special-cases are tracked so that we can call out to compiled routines that expect contiguous arrays more easily. So, your mistake is trying to think that the block of memory is [0, 3, 1, 4, 2, 5] when you pass the Fortran-order array to the reshape method.While this is true, and it means that you will save a copy if you passed this off to a Fortran routine, the reshape command does not use this information in determining how to think-about the input array. In fact, the reshape method does not allow any way to specify the order of the input array (self) separately from the order of the output array. The order argument indicates the defined order of both input and output. You might think that the order of self should be used as the order of the input array. The problem with this is, again, that a general array does not have a defined order. What should be used as the assumed order for an un-strided array? You're left with an unresolved question. To avoid two input order arguments, I just let order indicate the order for both the input and the output arrays. We could provide both, but this seems a bit over-done as the same could be accomplished by separately raveling the input to 1-d and then specifying the order argument on reshape. Please continue to question. All the code needs as much review as it can get. Best regards, -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] NumPy 1.0rc3 will be out in a few hours
I'm going to release NumPy 1.0rc3 in a few hours. It requires a re-build of extension modules from 1.0rc2. This gives us a chance to start fresh with NumPy 1.0 and also test the non-dying version of the version_checking code. So, it's not all bad. Official NumPy 1.0 will be released on Oct. 24 to many tears of rejoicing... Please file all tickets / requests by this weekend if you hope to have them included in the 1.0 release. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] questions regarding assignement and copy
David Cournapeau wrote: Sven Schreiber wrote: Yes it's intended; as far as I understand the python/numpy syntax, + is an operator, and that triggers assignment by copy (even if you do something trivial as bar = +foo, you get a copy, if I'm not mistaken), So basically, whenever you have foo = expr with expr is a numpy expression containing foo, you trigger a copy ? I think you are better off understanding that = is a name binding operation while += calls a special method that allows in-place adding. Thus, bar += foo calls bar.__iadd__(bar, foo) which gives the opportunity to add foo to bar in-place while bar = bar + foo adds bar to foo (which results in a new array that then gets re-bound to the name bar). The '=' sign is not an operator. Perhaps this will help you see the difference. It's good to know what kinds of things trip people up. We can target tutorials and FAQs to those things. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] The NumPy Fortran-ordering quiz
Charles R Harris wrote:
On 10/17/06, *Charles R Harris* [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED] wrote:
On 10/17/06, *A. M. Archibald* [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED] wrote:
On 17/10/06, Charles R Harris [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED] wrote:
On 10/17/06, Travis Oliphant [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED] wrote:
snip
Which doesn't seem to be the case here. I am beginning to wonder
if we really need fortran order, seems that a few well chosen
interface routines would fill the need and avoid much confusion.
For instance, it would be nice if flatten took an order keyword:
In [107]: array([[1,2,3],[4,5,6]], dtype=int8, order='F').flatten()
Out[107]: array([1, 2, 3, 4, 5, 6], dtype=int8)
It does take an argument (just not a keyword argument). The general
rule I followed (probably inappropriately) was that single-argument
methods didn't need keywords.
so
a.flatten('F') gives you a Fortran-order flattening.
-Travis
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] The NumPy Fortran-ordering quiz
Currently, the key operation is reshape, which only needs to return a
view in fortran order and doesn't even need to mark the resulting
array as fortran order because, well, because it works just fine in
numpy as is, it just isn't contiguous. If the other functions took
shape and order, reshape wouldn't even need the order keyword.
The flag is the there as a quick check for interfacing. The order
keyword grew because it was useful to avoid the arbitrariness of
C-contiguous order for those who prefer to think of it differently.
Remember the .T attribute for .transpose() was a recent addition and
sticking .transpose() everywhere is a lot more ugly. But, yes, many
uses of the order keyword could be replaced by preceding with
.transpose() --- this is not without cost, however.
I don't see why the array constructor needs the order keyword, it
doesn't *do* anything. For instance
a = array([[1,2,3],[4,5,6]], order='F')
doesn't produce a fortran contiguous array, it produces the same array
as the 'C' form, just sets the fortran flag and marks contiguous as
False. What is the use of that? It is just a generic non-contiguous
numpy array.
What? You're not understanding something. The order flag definitely
does something here. First of all it seems like you are not
understanding the meaning of the CONTIGUOUS flag. CONTIGUOUS means
C-order contiguous while FORTRAN means FORTRAN-order contiguous.
That's why I use the word single-segment to talk about FORTRAN-order or
C-contiguous order. For Numeric, CONTIGUOUS always meant C-order
contiguous and we are continuing that tradition. All we've done is
notice that there is such a think as FORTRAN-order contiguous and copies
do not need to be made in all circumstances when you have FORTRAN-order.
Look at the difference between:
a = array([[1,2,3],[4,5,6]],order='F').data[:]
b = array([[1,2,3],[4,5,6]]).data[:]
Notice the layout is definitely different between a and b.
And
In [131]: ascontiguousarray(array([[1,2,3],[4,5,6]], dtype=int8,
order='F')).flags
Out[131]:
CONTIGUOUS : True
FORTRAN : False
OWNDATA : True
WRITEABLE : True
ALIGNED : True
UPDATEIFCOPY : False
Doesn't produce a fortran contiguous array, so what use was the flag? And
Because you requested a C-contiguous array --- that's what contiguous
means in NumPy (exactly what it meant in Numeric).
In [141]: array([1,2,3,4,5,6], dtype=int8).reshape((2,3),
order='F').astype(int16).flags
Out[141]:
CONTIGUOUS : True
FORTRAN : False
OWNDATA : True
WRITEABLE : True
ALIGNED : True
UPDATEIFCOPY : False
reorders stuff in memory, so is a bug looking to happen in a fortran
interface.
Yes, like I said before, all kinds of operations alter the layout of
data. You can't assume all operations will preserve FORTRAN ordering.
FORTRAN-order has meaning beyond how the data is actually set out in
memory. Sometimes it indicates how you think it is layed out when you
are doing re-shaping operations.
mmapped files are the only thing I can think of where one might want
vary an operation depending on Fortran ordering because seeking out of
order is very expensive. But that means adapting algorithms depending
on order type, better I think to just stick to using the small strided
dimensions when appropriate.
It would be helpful in debugging all this order stuff if it was clear
what was supposed to happen in every case. Ravel, for instance,
ignores the FORTRAN flag, again begging the question as to why we
*have* the flag.
No it doesn't. Please show your evidence. Look:
a = array([[1,2,3],[4,5,6]])
print a.ravel()
[1 2 3 4 5 6]
print a.ravel('F')
[1 4 2 5 3 6]
If it's not working in some cases, please report that as a bug.
-Travis
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] The NumPy Fortran-ordering quiz
I'm not talking about the keyword in the ravel call, I'm talking about the flag in a. Ah. Yes, I see. I misunderstood. Of course ravel ignores the FORTRAN flag (actually it doesn't because if a copy is not necessary it doesn't make one).The key is that the Python user doesn't need to care about the array flag unless they are interfacing to compiled code. That's the point of the flag. It's actually redundant because it could be checked every time it's needed. But, right now, it's kept updated so that the check is simple. The same is true with the C-CONTIGUOUS flag (called contiguous). The question is: do we *need* a fortran flag. No, you don't *need* the flag. But, it saves copying data to check it (look how many times ISFORTRAN is called in the code). Without the flag all of those cases would need to do a strides-check which is done in the UpdateFlags code. I am argueing not, because the only need is for fortran contiguous arrays to pass to fortran function, or translation from fortran contiguous arrays to numpy arrays. What I am saying is that things are unnecessarily complicated. I disagree. It's actually not that complicated. Even if it was compilcated to implement, the point is that it is now done. There is no sense ripping it out (that would be a huge pain and for what purpose?) The FORTRAN flag gives us a lot more flexibility when it comes to copying data or not. I think part of the complication is that you are misunderstanding some of the terms and the purposes of the keywords. None of the LaPack stuff seems to use the Fortran stuff, they just transpose and copy. It doesn't now only because I haven't had time to go through and change it, but it should. Look at scipy's LaPack interface. It (through f2py) uses the FORTRAN stuff extensively (much was borrowed from there in the first place). -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] The NumPy Fortran-ordering quiz
Tim Hochberg wrote: One thing that may be confusing the issue is that, as I understand it, FORTRAN and CONTIGUOUS together represent three states which I'll call FORTRAN_ORDER, C_ORDER and DISCONTIGUOUS. Yep, that's what they mean. CONTIGUOUS is the name Numeric gave it and it meant C-order contiguous. We have kept the same meaning. All we've done is selected out from the class of arrays that Numeric called DISTCONTIGUOUS, arrays that are FORTRAN-order (and so still single-segment), but discontiguous in the sense that Numeric had. I periodically wonder if it would be valuable to have a way to query the order directly: the result would be C, F or None, just like the order keyword that is passed in. You an do it with the flags a.flags.contiguous a.flags.fortran Discontiguous is when both of these are false. Note that for a.ndim 2, both a.flags.contiguous and a.flags.fortran are true if one of them is true. This is all explained in the first chapters of my book. You have to understand CONTIGUOUS == C-order contiguous and FORTRAN == Fortran-order contiguous. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] What does Fortran order mean?
Charles R Harris wrote: Travis, I note that a = arange(6).reshape(2,3,order='F') a array([[0, 1, 2], [3, 4, 5]]) Shouldn't that be 3x2? Or maybe [[0,2,4],[1,3,5]]? Reshape is making a copy, but flat, flatten, and tostring all show the elements in 'C' order. I ask because I wonder if changing the order can be used to prepare arrays for input into the LaPack routines. The order argument to reshape means (how should the big-chunk of data be interpreted when you reshape). So, yes, this should be [[0,2,4],[1,3,5]]. It is a bug that it does not do the right thing in this case. I've committed a fix to SVN and a test for this case. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Accessing data buffers in numpy scalars
Francesc Altet wrote: Hi, I'm looking for an easy way to access the data area of the numpy scalars no matter its type. I've seen that numpy/arrayscalars.h define a structure for each scalar type, so I'd guess that it will not be possible to find a general way for accessing the data buffer for each type. So, I've decided to look for a workaround and I've devised a couple of possibilities: This problem showed up in writing NumPy several times. One solution might be to use PyArray_ScalarAsCtype(PyObject *scalar, void *ctypeptr) which will copy the data into the area pointed to by ctypeptr (unless you have a flexible scalar in which case only a pointer to the data-area will be copied). 2.- Fetch the buffer in scalartype.data and use the buffer protocol in order to access the pointer to data in memory. However, I lack experience in buffer protocol, so suggestions for achieving this are welcome. This will also work. A read-only buffer protocol is exported by all the scalars. scalar.data will return a buffer object. Or you can use the Python C-API const char *buffer; Py_ssize_t buflen; PyObject_AsReadBuffer(scalar, (const void **)buffer, buflen) to retrieve a pointer to the data in buffer and the size of the data in buflen. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] What does Fortran order mean?
Stefan van der Walt wrote: On Tue, Oct 17, 2006 at 10:01:51AM -0600, Travis Oliphant wrote: Charles R Harris wrote: Travis, I note that a = arange(6).reshape(2,3,order='F') a array([[0, 1, 2], [3, 4, 5]]) Shouldn't that be 3x2? Or maybe [[0,2,4],[1,3,5]]? Reshape is making a copy, but flat, flatten, and tostring all show the elements in 'C' order. I ask because I wonder if changing the order can be used to prepare arrays for input into the LaPack routines. The order argument to reshape means (how should the big-chunk of data be interpreted when you reshape). So, yes, this should be [[0,2,4],[1,3,5]]. It is a bug that it does not do the right thing in this case. A bit counter-ituitive (I somehow expect reshape to return an array that satisfies all the constraints specified as parameters -- i.e. shape and order in memory), but consistent with Numeric. Perhaps, but that is not what reshape does. When you are reshaping an array, unless you are just adding ones to the shape somewhere or aren't actually changing the shape, then you are implicitly thinking about the array as a 1-d chunk of memory in some order. The default is C-order. The order argument let's you think about it differently. What confuses me is that, if you call the array constructor, you get In [2]: N.array(N.array([[1,2,3],[4,5,6]]),order='F') Out[2]: array([[1, 2, 3], [4, 5, 6]]) so here 'order' means something else. Not really. I can't imagine what else you would expect here. You have specified the array explicitly. The order argument only specifies how the array will be arranged in memory. It is an implementation detail. The user is allowed some control in order to interface to compiled code more effectively. So then you do x = N.array([[0,1,2],[3,4,5]],order='F') x.reshape((2,3),order='C') and you get array([[0, 1, 2], [3, 4, 5]]) I don't see the problem. The first line gives you an array: [[0,1,2], [3,4,5]] which in memory is laid out as [0,3,1,4,2,5] The second line says reshape into a (2,3) array but this is exactly the same shape as the input array so in fact only a new view of the same data is given to you. The fact that you specify order is inconsequential since you aren't really changing the shape, it doesn't matter. The reshape command is not intended to alter the underlying ordering of an input array. The ordering of an array is an implementation detail that shouldn't matter to the user unless you are trying to interface to compiled code. The require command can help you get the right kind of array for that purpose. In general, arrays can have arbitrary strides. C-order and Fortran-order are just two special cases of that generality. So, you cannot think of any array as just being either in C-order or Fortran-order. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] array repr
Andrew MacKeith wrote: I would like to use the built-in array_repr in numpy, because I need a fast repr that does not contain new line characters. I see no way of doing this without editing the code in numeric.py, and I hate to edit other people's libraries. from numpy import array causes numeric.py to be executed, which includes a call to multiarray.set_string_function(array_repr, 1) If I want to undo this, there is no way of doing it. Why do want to undo this? I would like to unset the repr with: numpy.set_string_function(None, 1) I don't understand what you are trying to do. Why don't you just set the string function to an actual callable. What edits do you propose to make? -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] array repr
Ah!, I get it. You want to be able to reset to the C-defined array_repr function. The one that gets over-written on import.That makes sense. And is definitely do-able. Please file a ticket. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] matlab, filter function
Kenny Ortmann wrote:
just looking for some help, most of the time you guys are good with matlab
code, i am trying to use the filter function under this setting
y = filter(b,a,X) filters the data in vector X with the filter described by
numerator coefficient vector b and denominator coefficient vector a. If a(1)
is not equal to 1, filter normalizes the filter coefficients by a(1). If
a(1) equals 0, filter returns an error.
There is scipy.signal.lfilter which implements this algorithm. It's doc
string is
lfilter(b, a, x, axis=-1, zi=None)
Filter data along one-dimension with an IIR or FIR filter.
Description
Filter a data sequence, x, using a digital filter. This works for
many
fundamental data types (including Object type). The filter is a
direct
form II transposed implementation of the standard difference equation
(see Algorithm).
Inputs:
b -- The numerator coefficient vector in a 1-D sequence.
a -- The denominator coefficient vector in a 1-D sequence. If a[0]
is not 1, then both a and b are normalized by a[0].
x -- An N-dimensional input array.
axis -- The axis of the input data array along which to apply the
linear filter. The filter is applied to each subarray along
this axis (*Default* = -1)
zi -- Initial conditions for the filter delays. It is a vector
(or array of vectors for an N-dimensional input) of length
max(len(a),len(b)). If zi=None or is not given then initial
rest is assumed. SEE signal.lfiltic for more information.
Outputs: (y, {zf})
y -- The output of the digital filter.
zf -- If zi is None, this is not returned, otherwise, zf holds the
final filter delay values.
Algorithm:
The filter function is implemented as a direct II transposed
structure.
This means that the filter implements
y[n] = b[0]*x[n] + b[1]*x[n-1] + ... + b[nb]*x[n-nb]
- a[1]*y[n-1] + ... + a[na]*y[n-na]
using the following difference equations:
y[m] = b[0]*x[m] + z[0,m-1]
z[0,m] = b[1]*x[m] + z[1,m-1] - a[1]*y[m]
...
z[n-3,m] = b[n-2]*x[m] + z[n-2,m-1] - a[n-2]*y[m]
z[n-2,m] = b[n-1]*x[m] - a[n-1]*y[m]
where m is the output sample number and n=max(len(a),len(b)) is the
model order.
The rational transfer function describing this filter in the
z-transform domain is
-1 -nb
b[0] + b[1]z + ... + b[nb] z
Y(z) = -- X(z)
-1 -na
a[0] + a[1]z + ... + a[na] z
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] The NumPy Fortran-ordering quiz
Lisandro Dalcin wrote: I was surprised by this In [14]: array([[1,2,3],[4,5,6]]).reshape((3,2),order='F') Out[14]: array([[1, 5], [4, 3], [2, 6]]) In [15]: array([1,2,3,4,5,6]).reshape((3,2),order='F') Out[15]: array([[1, 2], [3, 4], [5, 6]]) This is a bug that was recently fixed in SVN. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] The NumPy Fortran-ordering quiz
Charles R Harris wrote: On 10/17/06, *Lisandro Dalcin* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I was surprised by this In [14]: array([[1,2,3],[4,5,6]]).reshape((3,2),order='F') Out[14]: array([[1, 5], [4, 3], [2, 6]]) This one still looks wrong. In [15]: array([1,2,3,4,5,6]).reshape((3,2),order='F') Out[15]: array([[1, 2], [3, 4], [5, 6]]) This one is fixed, In [3]: array([[1,2,3,4,5,6]]).reshape((3,2),order='F') Out[3]: array([[1, 4], [2, 5], [3, 6]]) I also don't understand why a copy is returned if 'F' just fiddles with the indices and strides; the underlying data should be the same, just the view changes. FWIW, I think both examples should be returning views. You are right, it doesn't need to. My check is not general enough. It can be challenging to come up with a general way to differentiate the view-vs-copy situation and I struggled with it. In this case, it's the fact that while self-nd 1, the other dimensions are only of shape 1 and so don't really matter. If you could come up with some kind of striding check that would distinguish the two cases, I would appreciate it. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Why doesn't array(a, dtype=single, copy=0) downcast a double array.
Charles R Harris wrote: It seems to me that since the behaviour when copy=0 is to make a copy only if necessary, it should find it necessary and make the downcast. After all, array(a, dtype=single, copy=1) does just that without complaint. Some common code in linalg could be replaced if array and asarray would do that operation. Well, the fact that it makes the copy without raising an error is different behavior from Numeric and should be considered an unintended change. We definitely should make this consistent for copy=0 or copy=1. The only question, is do we raise the error in both cases or allow the conversion in both cases. The long-standing behavior is to raise the error on possible-loss conversion and so my opinion is that we should continue with that behavior. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] How to make sqrt(-1) be 1j
Bill Baxter wrote: On 10/13/06, Tim Hochberg [EMAIL PROTECTED] wrote: For this sort of thing, I would just make a new module to pull together the function I want and use that instead. It's then easy to explain that this new module bbeconf (Bill Baxter's Excellent Collection Of Numeric Functions) is actually an amalgamation of stuff from multiple sources. # bbeconf.py from numpy import * fromnumpy.scimath import sqrt # possibly some other stuff to correctly handle subpackages... That does sound like a good way to do it. Then you just tell your users to import 'eduNumpy' rather than numpy, and you're good to go. Added that suggestion to http://www.scipy.org/NegativeSquareRoot I'd like to ask one basic Python question related my previous suggestion of doing things like numpy.sqrt = numpy.lib.scimath.sqrt: In python does that make it so that any module importing numpy in the same program will now see the altered sqrt function? E.g. in my program I do import A,B. Module A alters numpy.sqrt. Does that also modify how module B sees numpy.sqrt? If so then that's a very good reason not to do it that way. I've heard people using the term monkey-patch before. Is that what that is? --bb - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
David Goldsmith wrote: Travis Oliphant wrote: [EMAIL PROTECTED] wrote: Could sqrt(-1) made to return 1j again? Not in NumPy. But, in scipy it could. Ohmigod!!! You are definitely going to scare away many, many potential users - if I wasn't obliged to use open source at work, you'd be scaring me away. Why in the world does it scare you away. This makes no sense to me. If you don't like the scipy version don't use it. NumPy and SciPy are not the same thing. The problem we have is that the scipy version (0.3.2) already had this feature (and Numeric didn't). What is so new here that is so scary ? -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
David Goldsmith wrote: Travis Oliphant wrote: What could be simpler? ;-) Having sqrt(-1) return 1j (without having to remember that in order to get this, you have to write sqrt(-1+0j) instead). That's exactly what scipy.sqrt(-1) does. That was my point. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
I'd like to pitch in (again) on this issue, but I'll try to make sure that it's clear that I'm NOT arguing about sqrt() in particular, one way or another. Fernando, I don't disagree with you in principle. I don't think anybody does. I think we should try to keep the interfaces and expectations of scipy and numpy the same. Unfortunately, we have competing issues in this particular case (in the case of the functions in numpy.lib.scimath). Nobody has suggested an alternative to the current situation in SVN that is satsifying to enough users. Here is the situation. 1) NumPy ufuncs never up-cast to complex numbers without the user explicitly requesting it so sqrt(-1) creates a floating-point error condition which is either caught or ignored according to the user's desires.To get complex results from sqrt you have put in complex numbers to begin with. That's inherent in the way ufuncs work. This is long-standing behavior that has good reasons for it's existence. I don't see this changing.That's why I suggested to move the discussion over to scipy (we have the fancy functions in NumPy, they are just not in the top-level name-space). Now, it would be possible to give ufuncs a dtype keyword argument that allowed you to specify which underlying loop was to be used for the calculation. That way you wouldn't have to convert inputs to complex numbers before calling the ufunc, but could let the ufunc do it in chunks during the loop. That is certainly a reasonable enhancement: sqrt(a, dtype=complex). This no-doubt has a library-ish-feeling, but that is what NumPy is. If such a change is desireable, I don't think it would be much to implement it. 2) In SciPy 0.3.2 the top-level math functions were overloaded with these fancy argument-checking versions. So that scipy.sqrt(-1) would return 1j. This was done precisely to attract users like David G. who don't mind data-type conversions on the fly, but prefer automatic conversion (funny being called non-sensical when I was the one who wrote those original scipy functions --- I guess I'm schizophrenic). We changed this in SciPy 0.5.1 by accident without any discussion. It was simply a by-product of moving scipy_base (including those special-handling functions) into NumPy and forgetting to import those functions again into top-level SciPy.It was an oversight that caused backwards compatibilty issues. So, I simply changed it back to what SciPy used to be in SVN. If we want to change SciPy, then fine, but let's move that discussion over to scipy-dev. In short, I appreciate all the comments and the differences of opinion they point out, but they are ultimately non-productive. We can't just change top-level sqrt to be the fancy function willy-nilly. Paul says I'm nice (he's not talked to my children recently), but I'm not that much of a push-over. There are very good reasons that NumPy has the behavior it does. In addition, the fancy functions are already there in numpy in numpy.lib.scimath. So, use them from there if you like them. Create your own little mynumpy module that does from numpy import * from numpy.lib.scimath import * and have a ball. Python is flexible enough that the sky is not going to fall if the library doesn't do things exactly the way you would do it. We can still cooperate in areas that we agree on. Again. Put this to rest on NumPy and move the discussion over to scipy-dev. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
David Goldsmith wrote: I don't use scipy (and don't want to because of the overhead) but it sounds like I should because if I'm taking the square root of a variable whose value at run time happens to be real but less than zero, I *want* the language I'm using to return an imaginary; in other words, it's not the scipy behavior which scares me, its the numpy (which I do/have been using) behavior. O.K. Well the functions you want are in numpy.lib.scimath. I should have directed you there. You actually don't need scipy installed at all. Just import sqrt from numpy.lib.scimath. I'm sorry I misunderstood the issue. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
David Goldsmith wrote: Got it. And if I understand correctly, the import order you specify in the little mynumpy example you included in your latest response to Fernando will result in any overlap between numpy and numpy.lib.scimath to call the latter's version of things rather than the former's, yes? Right. The last import will be used for any common-names (variables get re-bound to the new functions...) -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Compiling Numpy with py2exe
Charles Hanot wrote: Hello, I’ve a program using Numpy and I would like to compile it with Py2exe but I’ve some problems. When I try to compile my program, some modules generated by numpy are not loaded. I’ve already asked to the py2exe community if they have a solution for that and the answer was that it was possible to compile numpy modules with py2exe but I should ask to the Numpy community J So my question is, is there something special to do in order to compile a program using numpy module? The only thing I remember is that you have to explicitly make sure _internal.py file gets copied over as it is not possible to discern that by looking at the import statements in .py files. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] multiarray tests skipped under py2.5
Tim Hochberg wrote: Still there with a clean checkout. Blech! I hate weird import problems. I'll try to look into it, but if anyone has any good ideas, let me know Show us the errors, I may be able to help. It's possible some recent changes messed things up. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] cannot import Numeric
Carl Wenrich wrote: thanks, but actually it's the other applications i want to use that have the 'import Numeric' line in them. i'm sure others have noted this before. what's the normal procedure? You must install Numeric if a package needs Numeric. As far as Python is concerned NumPy is a separate package. Packages must be ported to use numpy. Please encourage the package author to port. Help is available for open source packages. Just ask on the list. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Things to address for Py3K
Hi all, Py3K is undergoing active development. This gives us an opportunity to discuss more significant changes to the language that might improve the experience of NumPy users. We should form a list and start commenting on the py3k mailing lists about what changes would be most helpful for our community. Please provide examples of changes to Python that you think might help us. A couple on my short list 1) Adding a *few* new infix operators. a) an extra multiplication operator to distinguish between element-by-element and dot b) extending 'and' and 'or' to allow element-by-element logical operations or adding and || 2) Lowering the precedence of so that a 8 a 10 works as you would expect. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Things to address for Py3K
Christopher Barker wrote: Travis Oliphant wrote: A couple on my short list 1) Adding a *few* new infix operators. a) an extra multiplication operator to distinguish between element-by-element and dot b) extending 'and' and 'or' to allow element-by-element logical operations or adding and || 2) Lowering the precedence of so that a 8 a 10 works as you would expect. Maybe this goes without saying, but: 3) Inclusion of an nd-array type in the standard lib! (or at the very least, an nd-array protocol) Work on an nd-array protocol to extend the buffer protocol is occurring right now. It think this will be better in the end then a standard nd-array type. I think a multi-dimensional object array would at least be a nice step. There are enough differences between lists and 1-d arrays though, that I'm not sure the accepted multi-dimensional object array would just be the NumPy version. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
[EMAIL PROTECTED] wrote: Hi, I have recieved the following note from a user: In SciPy 0.3.x the ufuncs were overloaded by more intelligent versions. A very attractive feature was that sqrt(-1) would yield 1j as in Matlab. Then you can program formulas directly (e.g., roots of a 2nd order polynomial) and the right answer is always achieved. In the Matlab-Python battle in mathematics education, this feature is important. Now in SciPy 0.5.x sqrt(-1) yields nan. A lot of code we have, especially for introductory numerics and physics courses, is now broken. This has already made my colleagues at the University skeptical to Python as this lack of backward compatibility would never happen in Matlab. This was a consequence of moving scipy_base into NumPy but not exposing the scimath library in NumPy. It would be a very easy thing to put from numpy.lib.scimath import * into the scipy name-space. I'm supportive of that as a backward-compatibility measure. Another problem related to Numeric and numpy is that in these courses we use ScientificPython several places, which applies Numeric and will continue to do so. You then easily get a mix of numpy and Numeric in scripts, which may cause problems and at least extra overhead. Just converting to numpy in your own scripts isn't enough if you call up libraries using and returning Numeric. I wonder, what are the reasons that numpy.sqrt(-1) returns nan? Because that is backwards compatible. You have to construct a function-wrapper in order to handle the negative case correctly. The function wrapper is going to be slower. Thus, it is placed in a separate library. Could sqrt(-1) made to return 1j again? Not in NumPy. But, in scipy it could. If not, shouldn't numpy.sqrt(-1) raise a ValueError instead of returning silently nan? This is user adjustable. You change the error mode to raise on 'invalid' instead of pass silently which is now the default. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
Sven Schreiber wrote: This is user adjustable. You change the error mode to raise on 'invalid' instead of pass silently which is now the default. -Travis Could you please explain how this adjustment is done, or point to the relevant documentation. numpy.sqrt(-1) old = seterr(invalid='raise') numpy.sqrt(-1) # should raise an error seterr(**old) # restores error-modes for current thread numpy.sqrt(-1) - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
Stefan van der Walt wrote: I agree with Fernando on this one. Further, if I understand correctly, changing sqrt and power to give the right answer by default will slow things down somewhat. But is it worth sacrificing intuitive usage for speed? For NumPy, yes. This is one reason that NumPy by itself is not a MATLAB replacement. N.power(2,-2) == 0 and N.sqrt(-1) == nan just doesn't feel right. Only because your expectations are that NumPy *be* a MATLAB replacement. The problem is that it sacrifices too much for that to be the case. And we all realize that NumPy needs more stuff added to it to be like IDL/MATLAB such as SciPy, Matplotlib, IPython, etc. Why not then have N.power(2,-2) == 0.24 N.sqrt(-1) == 1j and write a special function that does fast calculation of square-roots for positive values? We've already done this. The special functions are called numpy.power numpy.sqrt (notice that if you do numpy.sqrt(-1+0j) you get the expected answer emphasizing that numpy does no argument checking to determine the output). The intuitive functions (which must do argument checking) are (in numpy.lib.scimath) but exported as scipy.power (actually I need to check that one...) scipy.sqrt What could be simpler? ;-) -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
[EMAIL PROTECTED] wrote: On Wed, 11 Oct 2006, Travis Oliphant wrote: On the other hand requiring all calls to numpy.sqrt to go through an argument-checking wrapper is a bad idea as it will slow down other uses. Interestingly, in worst cases numpy.sqrt is approximately ~3 times slower than scipy.sqrt on negative input but ~2 times faster on positive input: In [47]: pos_input = numpy.arange(1,100,0.001) In [48]: %timeit -n 1000 b=numpy.sqrt(pos_input) 1000 loops, best of 3: 4.68 ms per loop In [49]: %timeit -n 1000 b=scipy.sqrt(pos_input) 1000 loops, best of 3: 10 ms per loop This is the one that concerns me. Slowing everybody down who knows they have positive values just for people that don't seems problematic. In [50]: neg_input = -pos_input In [52]: %timeit -n 1000 b=numpy.sqrt(neg_input) 1000 loops, best of 3: 99.3 ms per loop In [53]: %timeit -n 1000 b=scipy.sqrt(neg_input) 1000 loops, best of 3: 29.2 ms per loop nan's are making things really slow, Yeah, they do. This actually makes the case for masked arrays, rather than using NAN's. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
Tim Hochberg wrote: With python 2.5 out now, perhaps it's time to come up with a with statement context manager. Something like: from __future__ import with_statement import numpy class errstate(object): def __init__(self, **kwargs): self.kwargs = kwargs def __enter__(self): self.oldstate = numpy.seterr(**self.kwargs) def __exit__(self, *exc_info): numpy.seterr(**self.oldstate) a = numpy.arange(10) a/a # ignores divide by zero with errstate(divide='raise'): a/a # raise exception on divide by zer # Would ignore divide by zero again if we got here. -tim This looks great. I think most people aren't aware of the with statement and what it can do (I'm only aware because of your posts, for example). So, what needs to be added to your example in order to just add it to numpy? -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] incrementing along a diagonal
David Novakovic wrote: Hi, i'm moving some old perl PDL code to python. I've come across a line which changes values in a diagonal line accross a matrix. matrix.diagonal() returns a list of values, but making changes to these does not reflect in the original (naturally). I'm just wondering if there is a way that i can increment all the values along a diagonal? You can refer to a diagonal using flattened index with a element-skip equal to the number of columns+1. Thus, a.flat[::a.shape[1]+1] += 1 will increment the elements of a along the main diagonal. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Should numpy.sqrt(-1) return 1j rather than nan?
Greg Willden wrote: On 10/11/06, *Travis Oliphant* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Stefan van der Walt wrote: Further, if I understand correctly, changing sqrt and power to give the right answer by default will slow things down somewhat. But is it worth sacrificing intuitive usage for speed? For NumPy, yes. This is one reason that NumPy by itself is not a MATLAB replacement. This is not about being a Matlab replacement. This is about correctness. I disagree. NumPy does the correct thing when you realize that sqrt is a function that returns the same type as it's input. The field over-which the operation takes place is defined by the input data-type and not the input values. Either way can be considered correct mathematically. As Paul said it was a design decision not to go searching through the array to determine whether or not there are negative numbers in the array. Of course you can do that if you want and that's what scipy.sqrt does. Numpy purports to handle complex numbers. Mathematically, sqrt(-1) is a complex number. Or, maybe it's undefined if you are in the field of real numbers. It all depends. Therefore Numpy *must* return a complex number. Only if the input is complex. That is a reasonable alternative to your specification. If Numpy doesn't return a complex number then it shouldn't pretend to support complex numbers. Of course it supports complex numbers, it just doesn't support automatic conversion to complex numbers.It supports complex numbers the same way Python supports them (i.e. you have to use cmath to get sqrt(-1) == 1j) People can look at this many ways without calling the other way of looking at it unreasonable. I don't see a pressing need to change this in NumPy, and in fact see many reasons to leave it the way it is. This discussion should move to the scipy list because that is the only place where a change could occur. -Travis. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] suggestions for slices of matrices
JJ wrote: Hello. I haven't been following development too closely lately, but I did just download and reinstall the current svn version. For what its worth, I would like to again suggest two changes: Suggestions are nice. Working code is better. Many ideas are just too difficult to implement (and still work with the system as it exists) and so never get done. I'm not saying these ideas fit into this category, but generally if a suggestion is not taken it's very likely seen in that light. -- If M is a nxm matrix and P and Z are nx1 (or 1xn) matrices, then it would be nice if we could write M[P==Z,:] to obtain all columns and only those rows where P==Z. This works already if p and z are 1-d arrays. That seems to be the only issue. you want this to work with p and z being 2-d arrays (i.e. matrices). The problem is there is already a defined behavior for this case that would have to be ignored (i.e. special-cased to get what you want). This could be done within the special matrix sub-class of course, but I'm not sure it is wise. Too many special cases make life difficult down the road. It is better to just un-think the ambiguity between 1-d and 2-d arrays that was inspired by Matlab and recognize a 1-d situation when you have it. But, that's just my opinion. I'm not dead-set against special-casing in the matrix object if enough matrix-oriented people are in favor of it. But, it would be a feature for a later NumPy (not 1.0). Likewise, for 1xm (or mx1) matrices U and V, it would be nice to be able to use M[P==Z,U==V]. Same issue as before + cross-product versus element-by-element. Also, it would be nice to use M[P==Z,U==2], for example, to obtain selected rows where matrix U is equal to a constant. Again. Form the cross-product using ix_(). -- It would be nice to slice a matrix by using M[[1,2,3],[3,5,7]], for example. You can get the cross-product using M[ix_([1,2,3],[3,5,7])]. This was a design choice and I think a good one. It's been discussed before. I believe this would help make indexing more user friendly. In my humble opinion, I think indexing is a weak spot in numpy. I'm sorry you see it that way. I think indexing is a strength of numpy. It's a little different then what you are used to with Matlab, perhaps, but it is much more general-purpose and capable (there is one weak-spot in that a certain boolean indexing operations uses more memory than it needs to, but that is a separate issue...). The Matlab behavior can always be created in a sub-class. Best regards, -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.0rc2: imag attribute of numpy.float64 causes segfault
Peter Bienstman wrote: This is on an AMD64 platform: Python 2.4.3 (#1, Sep 27 2006, 14:14:48) [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 Type help, copyright, credits or license for more information. import numpy print numpy.__version__ 1.0rc2 a = numpy.float64(1.0) a 1.0 a.real 1.0 a.imag Segmentation fault Thanks for the test. Fixed in SVN r3299 -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] tensor product
Charles R Harris wrote: Hmmm, I notice that there is no longer a tensor product. As it was the only one of the outer, kron bunch that I really wanted, l miss it. In fact, I always thought outer should act like the tensor product for the other binary operators too. Anyway, mind if I put it back? I'm not opposed to the idea, necessarily. But, when and where did this exist? I don't remember it. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] tensor product
Charles R Harris wrote: On 10/9/06, *Tim Hochberg* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: snip Is this not the same things as numpy.multiply.outer(a, b)? (as opposed to outer(a, b), which appears to pretend that everything is a vector -- I'm not sure what the point of that is). Hmmm, yes, multiply.outer does do that. I thought that outer was short for multiply.outer and that the behaviour had changed. So the question is why outer does what it does. Unfortunately, I don't know the answer to that. numpy.outer is the same as Numeric.outerproduct and does the same thing. I'm not sure of the reason behind it. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Numpy rc2 is out
Release Candidate 2.0 is now out. Thanks to all the great testing and fixes that were done between 1.0rc1 and 1.0rc2 The release date for NumPy 1.0 is Oct. 17. There will be a freeze on the trunk starting Monday Oct. 16 so any changes should be in by then. If significant changes are made then we will release 1.0rc3 on Oct. 17 and push the release date of NumPy 1.0 to Oct 24. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] repmat
Bill Baxter wrote: [There seem to have been some gmail delivery problems that prevented my previous mail on this subject from being delivered] I've proposed that we fix repmat handle arbitrary dimensions before 1.0. http://projects.scipy.org/scipy/numpy/ticket/292 I don't think this is particularly controversial, just I'm guessing no-one's found the time to look at my proposed fixes. And gmail/sourceforge issues haven't helped either. Thanks for reminding us again. I don't think this is too bad of a deal. I propose to move repmat(A, M,N) to matlib.py and replace it in numpy with a function named tile(a, reps) That is more general-purpose. If this is not acceptable, please speak up. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Vectorizing code, for loops, and all that
Travis Oliphant wrote: Albert Strasheim wrote: [1] 12.97% of function time [2] 8.65% of functiont ime [3] 62.14% of function time If statistics from elsewhere in the code would be helpful, let me know, and I'll see if I can convince Quantify to cough it up. Please run the same test but using x1 = N.random.rand(39,2000) x2 = N.random.rand(39,64,1) z1 = x1[:,N.newaxis,:] - x2 Very similar results to what I had previously: [1] 10.88% [2] 7.25% [3] 68.25% Thanks, I've got some ideas about how to speed this up by eliminating some of the unnecessary calculations going on outside of the function loop, but there will still be some speed issues depending on how the array is traversed once you get above a certain size. I'm not sure there anyway around that, ultimately, due to memory access being slow on most hardware. Well, I tried out my ideas and didn't get much improvement (8-10%). Then, I finally realized more fully that the slowness was due to the loop taking place over an axis which had a very large stride so that the memory access was taking a long time. Thus, instead of picking the loop axis to correspond to the axis with the longest dimension, I've picked the loop axis to be one with the smallest sum of strides. In this particular example, the speed-up is about 6-7 times... -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Problems with Numexpr and discontiguous arrays
Tim Hochberg wrote: That would be easy to do. Right now the opcodes should work correctly on data that is spaced in multiples of the itemsize on the last axis. Other arrays are copied (no opcode required, it's embedded at the top of interp_body lines 64-80). The record array case apparently slips through the cracks when we're checking whether an array is suitable to be used correctly (interpreter.c 1086-1103). It would certainly not be any harder to only allow contiguous arrays than to correctly deal with record arrays. Only question I have is whether the extra copy will overwhelm the savings of that operating on contiguous data gives. The thing to do is probably try it and see what happens. OK, I've checked in a fix for this that makes a copy when the array is not strided in an even multiple of the itemsize. I first tried copying for all discontiguous array, but this resulted in a large speed hit for vanilla strided arrays (a=arange(10)[::2], etc.), so I was more frugal with my copying. I'm not entirely certain that I caught all of the problematic cases, so let me know if you run into any more issues like this. There is an ElementStrides check and similar requirement flag you can use to make sure that you have an array whose strides are multiples of it's itemsize. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] flat indexing of object arrays
Martin Wiechert wrote: Hi list, when I try to assign a sequence as an element of an object array via flat indexing only the first element of the sequence is assigned: import numpy numpy.version.version '1.0rc1.dev3171' from numpy import * a = ndarray ((2,2), object) a.flat [2] = (1, 2, 3) a.flat [2] 1 a array([[None, None], [1, None]], dtype=object) Is this a feature? Wouldn't a naive user like me expect a.flat [2] == (1, 2, 3)? You are probably right.This should be changed. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Hello and my first patch
A. M. Archibald wrote: On 05/10/06, Greg Willden [EMAIL PROTECTED] wrote: On 10/5/06, Travis Oliphant [EMAIL PROTECTED] wrote: Perhaps that is the best way to move forward along with the work on a pylab super-package. That is exactly what I want. What is unsatisfactory about installing numpy+scipy+matplotlib? I've found they're generally pretty complete (except where no decent python alternative exists). In the end I want a nice collection of functions, logically organized, that let me analyze/filter/plot etc. etc. etc. The key for me is logically organized. There is a structure to it, but it's more organic because of the multiple contributors. weave should be in NumPy but nobody was willing to step up to maintain it a year ago. I may be willing to step up at this point. I would like to see weave in NumPy (maybe not the blitz libraries though...) I think a hybrid for weave / f2py / ctypes that allows inlining in multiple languages as well as automatic extension module generation for already-written code is in order. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory errors
Vikalpa Jetly wrote: I am reading a very large array (~9000,11000) of 1 byte image values. I need to change values in the array that meet a certain condition so I am running something like: b = numpy.where(a200,0,1) to create a new array with the changed values. However, I get a MemoryError everytime I try this. I have over 3gb of RAM on my machine (most of which is available). The process runs fine on smaller datasets. Is there a maximum array size that numpy handles? Any alternatives/workarounds? The MemoryError is a direct result when system malloc fails.Rather than use where with two scalars (you're resulting array will be int32 and therefore 4-times larger). Use b = zeros_like(a) b[a200] = 1 which will consume less memory. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Problems with Numexpr and discontiguous arrays
Tim Hochberg wrote: David M. Cooke wrote: On Wed, 04 Oct 2006 10:19:08 -0700 Tim Hochberg [EMAIL PROTECTED] wrote: Ivan Vilata i Balaguer wrote: It seemed that discontiguous arrays worked OK in Numexpr since r1977 or so, but I have come across some alignment or striding problems which can be seen with the following code:: I looked at this just a little bit and clearly this bit from interp_body cannot work in the presence of recor arrays: // intp sf1 = sb1 / sizeof(double);\ //... #define f1((double *)x1)[j*sf1] There are clearly some assumptions that sb1 is evenly divisible by sizeof(double). Blech!. This is likely my fault, and I expect it won't be too horrible to fix, but I don't know that I'll have time immediately. My thinking is that this should be handled by a copy, so that the opcodes always work on contiguous data. The copy can be another opcode. One advantage of operating on contiguous data is that it's easier to use the processor's vector instructions, if applicable. That would be easy to do. Right now the opcodes should work correctly on data that is spaced in multiples of the itemsize on the last axis. Other arrays are copied (no opcode required, it's embedded at the top of interp_body lines 64-80). The record array case apparently slips through the cracks when we're checking whether an array is suitable to be used correctly (interpreter.c 1086-1103). It would certainly not be any harder to only allow contiguous arrays than to correctly deal with record arrays. Only question I have is whether the extra copy will overwhelm the savings of that operating on contiguous data gives. With record arrays you have to worry about alignment issues. The most complicated part of the ufunc code is to handle that. The usual approach is to copy (and possibly byte-swap at least the axis you are working on) over to a buffer (the copyswapn functions will do that using a pretty fast approach for each data-type). This is ultimately how the ufuncs work (though the buffer-size is fixed so the data is copied and operated on in chunks). -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Hello and my first patch
Greg Willden wrote: Hello All, I introduced myself on the Scipy list and I have a feeling that most of the subscribers here are also on Scipy-devel. Anyway I just submitted my first patch to numpy (ticket #316). I added the blackman-harris, Nuttall and Flat Top windowing functions and added See also sections to the docstrings for all the window functions. Great contribution. Thanks a bunch. I think this will probably go into the scipy package, though. There is already a lot of windows available in the scipy.signal.window function. The window functions that are in NumPy are there only for historical purposes only (i.e. compatibility with old MLab). On the other hand, the other thought to consider is that since we have window functions in NumPy already. Let's just move them all from scipy.signal into NumPy. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Hello and my first patch
Sebastian Haase wrote: If scipy is going to be installable as separate sub-packages maybe all window functions can be moved to scipy ;-) In other words, if the ones in numpy are there only for historical reasons maybe they should be cleaned out before the 1.0 release. All arguments seem similar to ndimage (which was is numarray and is now in scipy) Not really, because these functions were in *both* Numeric and numarray. That's the trouble. And the multiple scipy packages situation needs more discussion We are all ears... -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Vectorizing code, for loops, and all that
Albert Strasheim wrote: [1] 12.97% of function time [2] 8.65% of functiont ime [3] 62.14% of function time If statistics from elsewhere in the code would be helpful, let me know, and I'll see if I can convince Quantify to cough it up. Please run the same test but using x1 = N.random.rand(39,2000) x2 = N.random.rand(39,64,1) z1 = x1[:,N.newaxis,:] - x2 Very similar results to what I had previously: [1] 10.88% [2] 7.25% [3] 68.25% Thanks, I've got some ideas about how to speed this up by eliminating some of the unnecessary calculations going on outside of the function loop, but there will still be some speed issues depending on how the array is traversed once you get above a certain size. I'm not sure there anyway around that, ultimately, due to memory access being slow on most hardware. If anyone has any ideas, I'd love to hear them.I won't be able to get to implementing my ideas until at least Friday (also when rc2 will be released). -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] RC2 to be released Friday
I'm going to have to put off release of rc2 for Friday. I'm just too busy right now. That might help us get some speed-ups into the NOBUFFER_UFUNCLOOP code as well. My speed-up ideas are: 1) Only keep track of 1 set of coordinates instead of self-nargs sets (1 for each iterator). 2) Keep track of the dataptr for each iterator in the bufptr array (so a copy isn't necessary) 3) Not increment the index on each iterator separately. All of these changes will be made directly in the NOBUFFER_UFUNCLOOP code. More generally, it would be nice to take these ideas and push them into other areas of the code --- perhaps through the multi-iterator that is already present. Probably, this will have to wait until 1.0.1 though. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] ValueError: object too deep for desired array
[EMAIL PROTECTED] wrote: Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp. C:\Documents and Settings\kennycd c:\lameness C:\lamenessc:\Python24\python.exe templatewindow.py a = [[ 1.00013175e+00 2.63483019e-05 1.6740e+00 5.22246363e-05 1.8735e+00 -2.77694969e-07 -1.30756273e-07 1.03202730e-06] [ 2.63483019e-05 6.95644927e-05 -7.15426839e-07 1.99534228e-05 5.29400631e-05 -3.07638369e-09 -5.52729618e-06 -7.61431767e-06] [ 1.6740e+00 -7.15426839e-07 1.00011917e+00 2.50407751e-05 1.6219e+00 -2.77757947e-07 -1.30856101e-07 1.23058301e-07] [ 5.22246363e-05 1.99534228e-05 2.50407751e-05 8.21505582e-05 5.26966037e-05 -6.13563429e-09 -2.76420755e-06 -3.80791858e-06] [ 1.8735e+00 5.29400631e-05 1.6219e+00 5.26966037e-05 1.00020132e+00 5.01389982e-05 3.45959412e-05 3.17129503e-05] [ -2.77694969e-07 -3.07638369e-09 -2.77757947e-07 -6.13563429e-09 5.01389982e-05 3.45959412e-05 3.17129503e-05 3.27035490e-05] [ -1.30756273e-07 -5.52729618e-06 -1.30856101e-07 -2.76420755e-06 3.45959412e-05 3.17129503e-05 3.27035490e-05 3.59732704e-05] [ 1.03202730e-06 -7.61431767e-06 1.23058301e-07 -3.80791858e-06 3.17129503e-05 3.27035490e-05 3.59732704e-05 4.12184645e-05]] F = [[ -1.44231014e+03] [ -7.54006917e-02] [ -1.44227222e+03] [ -7.49199956e-02] [ -1.44242446e+03] [ -4.24684780e-02] [ -2.49566072e-02] [ -2.16978637e-02]] Traceback (most recent call last): File C:\lameness\PlotPanel.py, line 476, in OnNo self.parent.peaktime() File C:\lameness\PlotPanel.py, line 64, in peaktime self._replot() File C:\lameness\PlotPanel.py, line 159, in _replot self.drawpeak() File C:\lameness\PlotPanel.py, line 233, in drawpeak COE[:,j1-1] = linalg.solve(A,F) ValueError: object too deep for desired array Anyone know what this error could be coming from? Is it because the floating point numbers are too large, memory wise? No, it's because you are trying to fit a 2-d array into a 1-d position. linalg.solve(A,F) produces a 2-d array in this instance (because F is a 2-d array). COE[:,j1-1:] =linalg.solve(A,F) should work or squeezing the result of the solution to a 1-d array would also work. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] return type diffences of indexed arrays with Intel C++ compiler (Python 2.5)
Lars Bittrich wrote: Hi all, recently I tried to compile numpy/scipy with Python 2.5 and Intel C++ compiler 9.1. At first everything was fine, but the scipy test produced a few errors. The reason was a little difference: numpy svn(1.0.dev3233) with Intel C compiler (Python 2.5): --- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:x[i] Out[4]:1.0 numpy svn(1.0.dev3233) with GCC 3.3 (Python 2.3): -- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:print x[i] Out[4]:array([1]) The Intel version gives me a scalar whereas the gcc version an array. Maybe Python 2.5 is causing this problem but my first guess was the compiler. This is a Python 2.5 issue (the new __index__ method) was incorrectly implemented and allowing a 1-d array to be interpreted as an index. This should be fixed in SVN. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Vectorizing code, for loops, and all that
Travis Oliphant wrote: I suspect I know why, although the difference seems rather large. [snip] I'm surprised the overhead of adjusting pointers is so high, but then again you are probably getting a lot of cache misses in the first case so there is more to it than that, the loops may run more slowly too. I'm personally bothered that this example runs so much more slowly. I don't think it should. Perhaps it is unavoidable because of the memory-layout issues. It is just hard to believe that the overhead for calling into the loop and adjusting the pointers is so much higher. But, that isn't the problem, here. Notice the following: x3 = N.random.rand(39,2000) x4 = N.random.rand(39,64,1) %timeit z3 = x3[:,None,:] - x4 10 loops, best of 3: 76.4 ms per loop Hmm... It looks like cache misses are a lot more important than making sure the inner loop is taken over the largest number of variables (that's the current way ufuncs decide which axis ought to be used as the 1-d loop). Perhaps those inner 1-d loops could be optimized (using prefetch or something) to reduce the number of cache misses on the inner computation, and the concept of looping over the largest dimension (instead of the last dimension) should be re-considered. Ideas, -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] ***[Possible UCE]*** Re: Vectorizing code, for loops, and all that
Albert Strasheim wrote:
Hello all
-Original Message-
From: [EMAIL PROTECTED] [mailto:numpy-
[EMAIL PROTECTED] On Behalf Of Travis Oliphant
Sent: 03 October 2006 02:32
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Vectorizing code, for loops, and all that
Travis Oliphant wrote:
I suspect I know why, although the difference seems rather large.
[snip]
I'm surprised the overhead of adjusting pointers is so high, but then
again you are probably getting a lot of cache misses in the first case
so there is more to it than that, the loops may run more slowly too.
I'm personally bothered that this example runs so much more slowly. I
don't think it should. Perhaps it is unavoidable because of the
memory-layout issues. It is just hard to believe that the overhead for
calling into the loop and adjusting the pointers is so much higher.
Firstly, thanks to Tim... I'll try his functions tomorrow.
Meanwhile, I can confirm that the NOBUFFER_UFUNCLOOP case in
PyUFunc_GenericFunction is getting exercised in the slower case. Here's some
info on what's happening, courtesy of Rational Quantify:
case NOBUFFER_UFUNCLOOP:
while (loop-index loop-size) {
for (i=0; iself-nargs; i++)
loop-bufptr[i] = loop-iters[i]-dataptr; [1]
loop-function((char **)loop-bufptr, (loop-bufcnt),
loop-steps, loop-funcdata); [2]
UFUNC_CHECK_ERROR(loop);
for (i=0; iself-nargs; i++) {
PyArray_ITER_NEXT(loop-iters[i]); [3]
}
loop-index++;
}
break;
[1] 12.97% of function time
[2] 8.65% of functiont ime
[3] 62.14% of function time
If statistics from elsewhere in the code would be helpful, let me know, and
I'll see if I can convince Quantify to cough it up.
Please run the same test but using
x1 = N.random.rand(39,2000)
x2 = N.random.rand(39,64,1)
z1 = x1[:,N.newaxis,:] - x2
Thanks,
-Travis
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-writeable default for numpy.ndarray
Tim Hochberg wrote: Francesc Altet wrote: It's not that the it's being built from ndarray, it's that the buffer that you are passing it is read only. This is correct. In fact, I'd argue that allowing the writeable flag to be set to True in this case is actually a bug. It's actually intentional. Strings used as buffers are allowed to be writeable. This is an explicit design decision to allow pickles to load without making 2 copies of the memory. The pickled string that Python creates is used as the actual memory for loaded arrays. Now, I suppose it would be possible to still allow this but be more finnicky about when a string-used-as-the-memory can be set writeable (i.e. we are the only reference to it). But, this would be a fragile solution as well. My inclination is to just warn users not to use strings as buffers unless they know what they are doing. The fact that it is read-only by default is enough of a guard against accidentally altering a string you didn't want to alter. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-writeable default for numpy.ndarray
Francesc Altet wrote: I see. Thanks for the explanation. You deserve the thanks for the great testing of less-traveled corners of NumPy. It's exactly the kind of thing needed to get NumPy ready for release. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] how to compile numpy with visual-studio-2003?
mg wrote: Hello, I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine on Linux-x86 32 and 64bit platforms. Now, I try to compile the both distributions on WindowsXP with VisualStudio2003. No problem to compile Python-2.5, but i have some troubles with Numpy-1.0rc1 and I didn't find any help in the provided setup.py. So, does someone can tell me how to do it? I don't use VisualStudio2003 on Windows to compile NumPy (I use mingw). Tim Hochberg once used a microsoft compiler to compile a previous version of NumPy and some things had to be fixed to make it work. I'm not sure if some in-compatibilities have crept in since then or not. But, I'd sure like to resolve it if they have. So, please post what problems you are having. You may be the first person to try a microsoft compiler with Python 2.5 -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
