Re: [Numpy-discussion] numpy.ctypeslib.ndpointer and the restype attribute [patch]
Thomas Heller wrote: Thomas Heller schrieb: Sturla Molden schrieb: On 3/26/2009 12:41 PM, Jens Rantil wrote: Wouldn't my code, or a tweak of it, be a nice feature in numpy.ctypeslib? Is this the wrong channel for proposing things like this? If you look at http://svn.scipy.org/svn/numpy/trunk/numpy/ctypeslib.py you will see that it does almost the same. I think it would be better to work out why ndpointer fails as restype and patch that. Thomas Heller schrieb: ndpointer(...), which returns an _nptr instance, does not work as restype because neither it is a base class of one of the ctypes base types like ctypes.c_void_p, also it is not callable with one argument. There are two ways to fix this. The first one is to make the _nptr callable [...] The other way is to make _nptr a subclass of ctypes.c_void_p, the result that the foreign function call returns will then be an instance of this class. Unfortunately, ctypes will not call __new__() to create this instance; so a custom __new__() implementation cannot return a numpy array and we are left with the _nptr instance. The only way to create and access the numpy array is to construct and return one from a method call on the _nptr instance, or a property on the _nptr instance. Ok, .errcheck could call that method and return the result. Well, looking into the ctypes sources trying to invent a new protocol for the restype attribute I found out that THERE IS ALREADY a mechanism for it, but I had totally forgotten that it exists. When the .restype attribute of a function is set to a SUBCLASS of a ctypes type (c_void_p for example), an instance of this subclass is created. After that, if this instance has a _check_retval_ method, this method is called and the result of this call is returned. So, it is indeed possible to create a class that can be assigned to .restype, and which can convert the return value of a function to whatever we like. I will prepare a patch for numpy.ctypeslib. It seems there isn't much interest in a patch - true? No, I'm very interested in the patch. Thanks for it. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] DVCS at PyCon
FYI from PyCon Here at PyCon, it has been said that Python will be moving towards DVCS and will be using bzr or mecurial, but explicitly *not* git. It would seem that *git* got the lowest score in the Developer survey that Brett Cannon did. The reasons seem to be: * git doesn't have good Windows clients * git is not written with Python I think the sample size was pretty small to be making decisions on (especially when most opinions where un-informed). I don't know if it matters that NumPy / SciPy use the same DVCS as Python, but it's a data-point. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is it a bug?
shuwj5...@163.com wrote: It's certainly weird, but it's working as designed. Fancy indexing via arrays is a separate subsystem from indexing via slices. Basically, fancy indexing decides the outermost shape of the result (e.g. the leftmost items in the shape tuple). If there are any sliced axes, they are *appended* to the end of that shape tuple. x = np.arange(30) x.shape = (2,3,5) idx = np.array([0,1,3,4]) e = x[:,:,idx] print e.shape #--- return (2,3,4) just as me think. e = x[0,:,idx] print e.shape #--- return (4,3). e = x[:,0,idx] print e.shape #--- return (2,4). not (4,2). why these three cases excute so # differently? This is probably best characterized as a wart stemming from a use-case oversight in the approach created to handle mixing simple indexing and advanced indexing. Basically, you can understand what happens by noting that when when scalars are used in combination with index arrays, they are treated as if they were part of an indexing array. In other words 0 is interpreted as [0] (or 1 is interpreted as [1]) when combined with advanced indexing. This is in part so that scalars will be broadcast to the shape of any indexing array to correctly handle indexing in other use-cases. Then, when advanced indexing is combined with ':' or '...' some special rules show up in determining the output shape that have to do with resolving potential ambiguities. It is arguable that the rules for resolving ambiguities are a bit simplistic and therefore don't handle some real use-cases very well like the case you show. On the other hand, simple rules are better even if the rules about combining ':' and '...' and advanced indexing are not well-known. So, to be a little more clear about what is going on, define idx2 = [0] and then ask what should the shapes of x[idx2, :, idx] and x[:, idx2, idx] be? Remember that advanced indexing will broadcast idx2 and idx to the same shape ( in this case (4,) but they could broadcast to any shape at all). This broadcasted result shape must be somehow combined with the shape resulting from performing the slice selection. With x[:, idx2, idx] it is unambiguous to tack the broadcasted shape to the end of the shape resulting from the slice-selection (i.e. x[:,0,0].shape). This leads to the (2,4) result. Now, what about x[idx2, :, idx]? The idx2 and idx are still broadcast to the same shape which could be any shape (in this particular case it is (4,)), but the slice-selection is done in the middle. So, where should the shape of the slice selection (i.e. x[0,:,0].shape) be placed in the output shape?At the time this is determined, there is no notion that idx2 came from a scalar and so it could have come from any array. Therefore, when there is this kind of ambiguity, the code always places the broadcasted shape at the beginning.Thus, the result is (4,) + (3,) -- (4.3). Perhaps it is a bit surprising in this particular case, but it is working as designed.I admit that this particular asymmetry does create some cognitive dissonance which leaves something to be desired. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is it a bug?
Robert Kern wrote: On Thu, Mar 12, 2009 at 01:34, Stéfan van der Walt ste...@sun.ac.za wrote: 2009/3/12 Robert Kern robert.k...@gmail.com: idx = np.array([0,1]) e = x[0,:,idx] print e.shape #- return (2,3). I think the right answer should be (3,2). Is # it a bug here? my numpy version is 1.2.1. It's certainly weird, but it's working as designed. Fancy indexing via arrays is a separate subsystem from indexing via slices. Basically, fancy indexing decides the outermost shape of the result (e.g. the leftmost items in the shape tuple). If there are any sliced axes, they are *appended* to the end of that shape tuple. This was my understanding, but now I see: In [31]: x = np.random.random([4,5,6,7]) In [32]: idx = np.array([1,2]) In [33]: x[:, idx, idx, :].shape Out[33]: (4, 2, 7) Hmm. Well, your guess is as good as mine at this point. Referencing my previous post on this topic. In this case, it is unambiguous to replace dimensions 1 and 2 with the result of broadcasting idx and idx together. Thus the (5,6) dimensions is replaced by the (2,) result of indexing leaving the outer dimensions in-tact, thus (4,2,7) is the result. I could be persuaded that this attempt to differentiate unambiguous from ambiguous sub-space replacements was mis-guided and we should have stuck with the simpler rule expressed above.But, it seemed so aesthetically pleasing to swap-out the indexed sub-space when it was possible to do it. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] suggestion for generalizing numpy functions
Darren Dale wrote: On Mon, Mar 9, 2009 at 9:50 AM, Darren Dale dsdal...@gmail.com mailto:dsdal...@gmail.com wrote: I spent some time over the weekend fixing a few bugs in numpy that were exposed when attempting to use ufuncs with ndarray subclasses. It got me thinking that, with relatively little work, numpy's functions could be made to be more general. For example, the numpy.ma http://numpy.ma module redefines many of the standard ufuncs in order to do some preprocessing before the builtin ufunc is called. Likewise, in the units/quantities package I have been working on, I would like to perform a dimensional analysis to make sure an operation is allowed before I call a ufunc that might change data in place. The suggestions behind this idea are interesting. It seems related to me, to the concept of contexts that Eric presented at SciPy a couple of years ago that keeps coming up at Enthought.It may be of benefit to solve the problem from that perspective rather than the sub-class perspective. Unfortunately, I don't have time to engage this discussion as it deserves, but I wanted to encourage you because I think there are good ideas in what you are doing.The sub-class route may be a decent solution, but it also might be worthwhile to think from the perspective of contexts as well. Basically, the context idea is that rather than sub-class the ndarray, you create a more powerful name-space for code that uses arrays to live in. Because python code can execute using a namespace that is any dictionary-like thing, you can create a namespace object with more powerful getters and setters that intercepts the getting and setting of names as the Python code is executing. This allows every variable to be adapted in a manner analagous to type-maps in SWIG --- but in a more powerful way.We have been taking advantage of this basic but powerful idea quite a bit. Unit-handling is a case where contexts and generic functions rather than sub-classes appears to be an approach to solving the problem. The other important idea about contexts is that you can layer-on adapters on getting and setting variables into the namespace which provide more hooks for doing some powerful things in easy-to-remember ways. I apologize if it sounds like I'm hi-jacking your question to promote an agenda. I really like the generality you are trying to reach with your suggestions and just wanted to voice the opinion that it might be better to look for a solution using the two dimensions of objects and namespaces (o.k. generic functions are probably another dimension in my metaphor) rather than just sub-classes of objects. -- Travis Oliphant Enthought, Inc. (512) 536-1057 (office) (512) 536-1059 (fax) http://www.enthought.com oliph...@enthought.com ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] NumPy SVN?
Is commit to NumPy SVN still turned off? How do I get a working SVN again? -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 64-bit numpy questions?
Todd Miller wrote:
Hi,
I've been looking at a 64-bit numpy problem we were having on Solaris:
a=numpy.zeros(0x18000,dtype='b1')
a.data
Traceback (most recent call last):
File stdin, line 1, in module
ValueError: size must be zero or positive
A working fix seemed to be this:
Index: arrayobject.c
===
--- arrayobject.c(revision 6530)
+++ arrayobject.c(working copy)
@@ -6774,7 +6774,7 @@
static PyObject *
array_data_get(PyArrayObject *self)
{
-intp nbytes;
+Py_ssize_t nbytes;
if (!(PyArray_ISONESEGMENT(self))) {
PyErr_SetString(PyExc_AttributeError, cannot get single-\
segment buffer for discontiguous array);
@@ -6782,10 +6782,10 @@
}
nbytes = PyArray_NBYTES(self);
if PyArray_ISWRITEABLE(self) {
-return PyBuffer_FromReadWriteObject((PyObject *)self, 0, (int)
nbytes);
+return PyBuffer_FromReadWriteObject((PyObject *)self, 0,
(Py_ssize_t) nbytes);
}
else {
-return PyBuffer_FromObject((PyObject *)self, 0, (int) nbytes);
+return PyBuffer_FromObject((PyObject *)self, 0, (Py_ssize_t)
nbytes);
}
}
This fix could be simpler but still illustrates the typical problem:
use of (or cast to) int rather than something pointer sized.
This looks like a problem with the port to Python2.5 not getting all the
Python C-API changes.There is no need to change the
intp nbytes
line, but the un-necessary casting to (int) in the calls to the
PyBuffer_ should absolutely be changed at least for Python 2.5 and above.
-Travis
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Creating arrays with 'titles' in dtype causes TypeError on data access
Ralph Heinkel wrote:
However here something is strange:
arr = array([('john', 4),('mark', 3)],
dtype=[(('source:yy', 'name'),'O'),(('source:xx','id'),int)])
arr[0]
('john', 4)
arr[0][0]
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: function takes at most 2 arguments (3 given)
Any ideas what I'm doing wrong? Any help would be appreciated.
This is a bug in NumPy. I will fix it in the trunk tonight.
-Travis
--
Travis Oliphant
Enthought, Inc.
(512) 536-1057 (office)
(512) 536-1059 (fax)
http://www.enthought.com
oliph...@enthought.com
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] views and object lifetime
Neal Becker wrote: How is it ensured, at the C api level, that when I have an array A, and a view of it B, that the data is not destroyed until both A and B are? One array, A, owns the data and will deallocate it only when its reference-count goes to 0.The view, B, has a reference to A (stored in the base attribute) and has OWNDATA set to false so that its deallocator simply decreases the reference count on the array, A, that actually owns the data. In the code look at the `array_dealloc` function in arrayobject.c and the base and OWNDATA flag-bit in the array-structure for details. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] FYI: New select-multiple-fields behavior
Hi all, As of r6358, I checked in the functionality to allow selection by multiple fields along with a couple of tests. ary['field1', 'field3'] raises an error ary[['field1', 'field3']] is the correct spelling and returns a copy of the data in those fields in a new array. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FYI: New select-multiple-fields behavior
Stéfan van der Walt wrote: Hi Travis 2009/2/12 Travis E. Oliphant oliph...@enthought.com: ary['field1', 'field3'] raises an error ary[['field1', 'field3']] is the correct spelling and returns a copy of the data in those fields in a new array. Is there absolutely no way of returning the result as a view? Not that I can think of --- it does match advanced indexing semantics to have it be a copy. -Travis -- Travis Oliphant Enthought, Inc. (512) 536-1057 (office) (512) 536-1059 (fax) http://www.enthought.com oliph...@enthought.com ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Selection of only a certain number of fields
Francesc Alted wrote:
A Saturday 07 February 2009, Neil escrigué:
Travis E. Oliphant oliphant at enthought.com writes:
I've been fairly quiet on this list for awhile due to work and
family schedule, but I think about how things can improve
regularly.One feature that's been requested by a few people is
the ability to select multiple fields from a structured array.
Thus, suppose *arr* is a structured array with dtype:
[('name', 'S25'),
('height', float),
('age', int),
('gender', 'S8')
]
Then, newarr = arr[['name', 'age']] should be a structured array
with just the name and age fields.
What are some common use cases for this feature?
I use structured arrays quite a lot, but I haven't found myself
wanting something like this. If I do need a subset of a structured
array generally I use something like
[rec[n] for n in 'name age gender'.split()]
Good point. However, there are still some very valid reasons for having
an idiom like:
newarr = arr[['name', 'age']]
returning a record array.
The first one (and most important IMO), is that newarr continues to be
an structured array (BTW, when changed this name from the original
record array?),
To avoid confusion with the record array subclass which maps
attributes to fields, Eric Jones and I have been using this terminology
for about a year.
-Travis
--
Travis Oliphant
Enthought, Inc.
(512) 536-1057 (office)
(512) 536-1059 (fax)
http://www.enthought.com
oliph...@enthought.com
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy and bitwise arrays?
Muhammad Alkarouri wrote: --- On Wed, 21/1/09, Stéfan van der Walt ste...@sun.ac.za wrote: From: Stéfan van der Walt ste...@sun.ac.za ... You can also take a look at Ilan Schnell's bitarray: http://pypi.python.org/pypi/bitarray/ Looks good to me. Thanks for the suggestion. You might also make use of the NumPy functions: packbits unpackbits fromfile Read the bits in as uint8 data using fromfile. Then, you can manipulate them either using bit twiddling or with indexing operations after unpacking to boolean arrays. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Trying to implement the array interface
Mark Asbach wrote: Hi there, I'm currently extending the Python wrapper for the Open Computer Vision Library (opencv) with the goal to interface numerical libraries as seemless as possible. Unfortunately, it doesn't seem to be that easy ;-) What I've done so far: - Added an __array_interface__ property to the Python side of OpenCV data structures (matrices and images) that uses version 3 of the interface definition and supplies the keys 'version', 'shape', 'typestr', 'data' and in some cases 'strides' when we have non-continuos memory layout. I think, I'm compatible to http://numpy.scipy.org/array_interface.shtml . Great. This is a good first step. - Added parsing the __array_interface__ of Python objects passed to OpenCV methods. I'm a bit unsure of how to use the C/C++ side (array struct) and if I can expect it to be there (for example: I don't provide one with OpenCV). Since I intend to keep OpenCV independent of numpy, calling functions from numpy.h is not an option, as far as I can see. You can't expect the __array_struct__ property to be there, but if it is, it allows you to get all the information you need with one attribute lookup (albeit in C) instead of many. The stuff described above is in the head revision of OpenCV, accessible via svn co https://opencvlibrary.svn.sourceforge.net/svnroot/opencvlibrary/trunk/opencv;. I've tried using the following packages with OpenCV this way: - numpy (1.0.4): everything works as expected. This is the most important library for OpenCV users, so this is a good sign. - pylab/matplotlib (0.91.2): seems to use numpy / scipy-core. Everything okay. - PIL (1.1.6): the array interface (Python side) doesn't adhere to the definition - no 'version' key, 'data' is string, not a tuple holding the pointer. What to do with this? That is probably true. I've worked a bit with PIL to get things to work, but haven't followed the project lately to see where it is at. One difficulty is that the PIL memory layout can be quite different from a NumPy array, and so that would be why the data is a string. The best source for implementing consuming of the interface is to look in the NumPy source code and look for where it grabs the __array_interface__ and/or __array_struct__ attribute and makes use of the data found there. - Numeric (24.2): I can create arrays from OpenCV datatypes and print them. Converting to other types however always yields 'Cannot convert scalar to float' or 'a float is required'. Strange :-/ Numeric.array instances also don't carry an __array_interface__. I can however convert by using numpy.arrays as intermediate step. I believe the __array_struct__ property was used instead. You can implement either __array_struct__ or __array_interface__ or both as an exporter. Thus, a consumer that wants to see everything has to consume both. I'm not sure I understand the error you are getting exactly. - Gnuplot (1.7): uses Numeric, so doesn't work as well - pymat: didn't check. Seems to use Numeric, test results cover Numeric 23 and Matlab 6.5 only, so this package might be dead? I don't remember pymat very well. - numarray: didn't check. Is there still any relevance of this package? A few tools still use it, but it is deprecated (as is Numeric of course). I'm glad to see you using the __array_interface__ because it will allow you to share memory with numXXX arrays. I'm sure you are also aware of the new buffer protocol in Python 2.6 and Python 3.0. This is the approach to take for the future. NumPy, of course, will support the __array_interface__ and __array_struct__ properties for some time. I'm hoping that a new release of NumPy will also support the new Python buffer interface. But, as of yet, it does not. Best regards, -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memmap dtype issue
Wim Bakker wrote: For a long time now, numpy's memmap has me puzzled by its behavior. When I use memmap straightforward on a file it seems to work fine, but whenever I try to do a memmap using a dtype it seems to gobble up the whole file into memory. I don't understand your question. From my experience, the memmap is working fine. Please post and example that illustrates your point. This, of course, makes the use of memmap futile. I would expect that the result of such an operation would give me a true memmap and that the data would be converted to dtype on the fly. There is no conversion on the fly when you use memmap. You construct an array of the same data-type as is in the file and then manipulate portions of it as needed. Am I doing something wrong? Are my expectations wrong? My guess is that your expectations are not accurate, but example code would help sort it out. Best regards, -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ironclad v0.7 released (NumPy on IronPython)
William Reade wrote: Hi all Hopefully someone here will be interested in this, and it won't be considered too spammy... please let me know if this isn't welcome, and I'll desist in future. I welcome these announcements, so my opinion is that you continue. Thanks for the work. It's great to see a path for running C extensions on IronPython. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANNOUNCE: EPD with Py2.5 version 4.0.30002 RC2 available for testing
Hello, We've recently posted the beta1 build of EPD (the Enthought Python Distribution) with Python 2.5 version 4.1.30001 to the EPD website. You may download the beta from here: http://www.enthought.com/products/epdearlyaccess.php You can check out the release notes here: https://svn.enthought.com/epd/wiki/Python2.5.2/4.1.300/Beta1 Please help us test it out and provide feedback on the EPD Trac instance: https://svn.enthought.com/epd or via e-mail to [EMAIL PROTECTED] If everything goes well, we are planning a final release for December. About EPD - The Enthought Python Distribution (EPD) is a kitchen-sink-included distribution of the Python™ Programming Language, including over 60 additional tools and libraries. The EPD bundle includes NumPy, SciPy, IPython, 2D and 3D visualization, database adapters, GUI building libraries, and a lot of other tools right out of the box. http://www.enthought.com/products/epd.php It is currently available as a single-click installer for Windows XP (x86), Mac OS X (a universal binary for OS X 10.4 and above), and RedHat 3 and 4 (x86 and amd64). EPD is free for academic use. An annual subscription and installation support are available for individual commercial use. Enterprise subscriptions with support for particular deployment environments are also available for commercial purchase. Enthought Build Team ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] More loadtxt() changes
John Hunter wrote: On Tue, Nov 25, 2008 at 12:16 PM, Pierre GM [EMAIL PROTECTED] wrote: A la mlab.csv2rec ? It could work with a bit more tweaking, basically following John Hunter's et al. path. What happens when the column names are unknown (read from the header) or wrong ? Actually, I'd like John to comment on that, hence the CC. More generally, wouldn't be useful to push the recarray manipulating functions from matplotlib.mlab to numpy ? Yes, I've said on a number of occasions I'd like to see these functions in numpy, since a number of them make more sense as numpy methods than as stand alone functions. John and I are in agreement here. The issue has remained somebody stepping up and doing the conversions (and fielding the questions and the resulting discussion) for the various routines that probably ought to go into NumPy. This would be a great place to get involved if there is a lurker looking for a project. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] More loadtxt() changes
Pierre GM wrote: OK then, I'll take care of that over the next few weeks... Thanks Pierre. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PIL.Image.fromarray bug in numpy interface
Jim Vickroy wrote: Hello, While using the PIL interface to numpy, I rediscovered a logic error in the PIL.Image.fromarray() procedure. The problem (and a solution) was mentioned earlier at: * http://projects.scipy.org/pipermail/numpy-discussion/2006-December/024903.html There does not seem to be a formal way to report errors to the PIL project, and I was told that the PIL/numpy interface was contributed by the numpy developers so I'm reporting it here. Please let me know if there is something additional I should do. I would suggest making a patch and submitting it to the PIL. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Old-style classes in tests
Tom Wright wrote: I am currently working on the Ironclad project porting numpy to Ironpython. It would be quite useful for me if HermitianTestCase in test_linalg.py was a new style-class instead of an old-style class - since Ironpython has a bug where dir operations do not work for classes inheriting from both old- and new- style classes and I'd very much prefer not to patch my version of numpy. In general, it would be useful if whenever this multiple inheritence pattern is used new-style classes are used rather than old style classes. This would require the following classes to change: test_numerictypes - create_values, read_values_plain, read_values_nested test_print - create_zeros, create_values, assign_values, byteorder_values test_io - Roundtriptest test_linalg - LinalgTestCase, HermitianTestCase I have no trouble making all classes new-style. +1 -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal for changing the names of inverse trigonometrical/hyperbolic functions
Francesc Alted wrote: So, IMHO, I think it would be better to rename the inverse trigonometric functions from ``arc*`` to ``a*`` prefix. Of course, in order to do that correctly, one should add the new names and add a ``DeprecationWarning`` informing that people should start to use the new names. After two or three NumPy versions, the old function names can be removed safely. What people think? +1 -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Breaking up the umathmodule file.
Charles R Harris wrote: Hi All, I propose: umath_funcs_c99.inc.src umath_funcs.inc.src umath_loops.inc.src umath_object.inc umathmodule.c This sounds fine to me. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array serialization with JSON
Christopher Barker wrote: Matthieu Brucher wrote: Last time I checked, repr() does the same thing as str(): the middle of the array may not be displayed... right. darn -- is that controllable? numpy.set_printoptions(threshold=1000) -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New ufuncs
Charles R Harris wrote: On Sun, Nov 9, 2008 at 11:29 PM, T J [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: On Thu, Nov 6, 2008 at 3:01 PM, T J [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: On Thu, Nov 6, 2008 at 2:36 PM, Charles R Harris [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I could add exp2, log2, and logaddexp2 pretty easily. Almost too easily, I don't want to clutter up numpy with a lot of functions. However, if there is a community for these functions I will put them in. I worry about clutter as well. Note that scipy provides log2 and exp2 already (scipy.special). So I think only logaddexp2 would be needed and (eventually) logdotexp2. Maybe scipy.special is a better place than in numpy? Then perhaps the clutter could be avoidedthough I'm probably not the best one to ask for advice on this. I will definitely use the functions and I suspect many others will as well---where ever they are placed. Since no one commented further on this, can we go ahead and add logaddexp2? Once in svn, we can always deal with 'location' later---I just don't want it to get forgotten. __ The functions exp2 and log2 are part of the C99 standard, so I'll add those two along with log21p, exp21m, and logaddexp2. The names log21p and exp21p look a bit creepy so I'm open to suggestions. I think the C99 standard is a good place to draw the line. We can put other ufuncs in scipy.special -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding the ability to clone a few fields from a data-type
I'm not sure what this accomplishes. Would the dummy fields that fill
in the space be inaccessible? E.g. tuple(subarr[i,j,k]) gives a tuple
with no numpy.void scalars? That would be a novel feature, but I'm not
sure it fits the problem. On the contrary:
Yes, that was the idea. You can do it now, but only in C. The real
problem right now from my point of view is that there is no way to tell
the dtype constructor to pad the itemsize to x bytes.If that were
changed, then many things would be possible.
OTOH, now that I think about it, I don't think there is really any
coherent way to mix field selection with any other indexing
operations. At least, not within the same brackets. Hmm. So maybe the
link to fancy indexing can be ignored as, ahem, fanciful.
Yeah, I was wondering how to do it well, myself, and couldn't come up
with anything which is why I went the .view route with another dtype.
By inaccessible and invisible dtype do you mean something like the
basic built-in void data type, but which doesn't try to report itself
when the dtype prints?
That sounds interesting but I'm not sure it's necessary because the
field specification can already skip bytes (just not bytes at the end
--- which is what I would like to fix).Perhaps what is needed is a
pseudo-dtype (something like 'c' compared to 'S1') which doesn't
actually create a new dtype but which is handled differently when the
dtype is created with the [('field1', type), ('field2', type2)]
approach. Specifically, it doesn't add an entry to the fields
dictionary nor an entry to the names but does affect the itemsize of the
element (and the offset of follow-on fields).
So, let's assume the character is 'v':
If we have an array with underlying dtype:
od = [('date', 'S10'), ('high', 'f4'), ('low', 'f4'), ('close', 'f4'),
('volume', 'i4')]
Then, we could define a new dtype
nd = [('date', 'S10'), ('', 'v8'), ('close', 'f4'), ('', 'v4')]
and arr.view(nd) would provide a view of the array where element
selection would be a tuple with just the date and close elements but the
itemsize would be exactly the same but nd.names would be ['date', 'close']
I like this approach. It impacts the API the very least but provides
the desired functionality.
-Travis
-Travis
Overall, I guess, I would present the feature slightly differently.
Provide a kind of inaccessible and invisible dtype for implementing
dummy fields. This is useful in other places like file parsing. At the
same time, implement a function that uses this capability to make
views with a subset of the fields of a structured array. I'm not sure
that people need an API for replacing the fields of a dtype like this.
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Adding the ability to clone a few fields from a data-type
Hi all, I'd like to add to NumPy the ability to clone a data-type object so that only a view fields are copied over but that it retains the same total size. This would allow, for example, the ability to select out a few records from a structured array using subarr = arr.view(cloned_dtype) Right now, it is hard to do this because you have to at least add a dummy field at the end. A simple method on the dtype class (fromfields or something) would be easy to add. It was thought in the past to do this with indexing arr['field1', 'field2'] And that would still be possible (and mostly implemented) if this feature is added. Thoughts? -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] creating a Numeric array from a numpy array LEAKS memory
Jose Borreguero wrote: Dear numpy users, I need to pass a Numeric array to some oldie code from a numpy array. I decided to go like this: for i in range(BIGNUMER): my_numpy_array=grabArray(i) na=Numeric.array( my_numpy_array, Numeric.Float) oldie_code(na) The constructor line: na=Numeric.array( my_numpy_array, Numeric.Float) does leak memory. Is there a way to pass the Numeric array to oldie_code without the leaks? This should work without memory leaks, but there may be a bug in NumPy or Numeric. Which version of Numeric and NumPy do you have? -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy CAPI questions
Lane Brooks wrote: I am using the numpy CAPI to write an extension module that returns a numpy Array from an imaging data source. I collect the image into a buffer that I allocate. I then create numpy Array using the PyArray_New(..) function and pass it the buffer. I then set the NPY_OWNDATA flag on the Array because I want the Array to deallocate the buffer when it is deleted. Is that the correct way to do it? The code snippet below is what I wrote, and it seems to be working just fine, but I wanted to verify that I am doing things correctly. NPY_OWNDATA means the object will try to deallocate the memory (make sure it was allocated with the same allocator as NumPy uses). Otherwise, you will need to set up another approach as I showed in my blog posting several weeks ago. Also, don't use Py_BuildValue with O as it will create another reference so that img will have an extra reference to it when it is returned to Python. Use N instead. However, in this case you don't need to use Py_BuildValue at all because you are returning only one array. The PyArray_UpdateFlags call is not used for changing NPY_OWNDATA. It is only useful for changing FORTRAN, CONTIGUOUS, ALIGNED, and WRITEABLE flags which are convenience flags. This call does the check first and then sets the state of the flag to reflect the actual situation for the array. Instead use PyArray_FLAGS(arr) |= NPY_OWNDATA; -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] choose() broadcasting
Roman Bertle wrote: * Travis E. Oliphant [EMAIL PROTECTED] [081003 22:20]: Roman Bertle wrote: Hello, I have found something I call a bug in the numpy choose() method and wanted to report it in trac. Thanks for your report. I'm not sure why you are having trouble with Trac, but I've created a ticket for this problem. Hello, trac works for me know. And thank you for fixing the bug, the svn numpy version works now for me. But there remains an issue I want to report. choose is much slower in numpy than in numarray, and even more if an output array is specified, as these tests show: Thanks for the report. You should add another ticket for this case. I suspect it might be a result of the extra copies that are done in the PyArray_Choose routine because the algorithm assumes contiguous arrays. It deserves a look. It probably wouldn't be too difficult to avoid the copy. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] var bias reason?
Gabriel Gellner wrote: Some colleagues noticed that var uses biased formula's by default in numpy, searching for the reason only brought up: http://article.gmane.org/gmane.comp.python.numeric.general/12438/match=var+bias which I totally agree with, but there was no response? Any reason for this? I will try to respond to this as it was me who made the change. I think there have been responses, but I think I've preferred to stay quiet rather than feed a flame war. Ultimately, it is a matter of preference and I don't think there would be equal weights given to all the arguments surrounding the decision by everybody. I will attempt to articulate my reasons: dividing by n is the maximum likelihood estimator of variance and I prefer that justification more than the un-biased justification for a default (especially given that bias is just one part of the error in an estimator).Having every package that computes the mean return the un-biased estimate gives it more cultural weight than than the concept deserves, I think. Any surprise that is created by the different default should be mitigated by the fact that it's an opportunity to learn something about what you are doing.Here is a paper I wrote on the subject that you might find useful: https://contentdm.lib.byu.edu/cdm4/item_viewer.php?CISOROOT=/EERCISOPTR=134CISOBOX=1REC=1 (Hopefully, they will resolve a link problem at the above site soon, but you can read the abstract). I'm not trying to persuade anybody with this email (although if you can download the paper at the above link, then I am trying to persuade with that). In this email I'm just trying to give context to the poster as I think the question is legitimate. With that said, there is the ddof parameter so that you can change what the divisor is. I think that is a useful compromise. I'm unhappy with the internal inconsistency of cov, as I think it was an oversight. I'd be happy to see cov changed as well to use the ddof argument instead of the bias keyword, but that is an API change and requires some transition discussion and work. The only other argument I've heard against the current situation is unit testing with MATLAB or R code. Just use ddof=1 when comparing against MATLAB and R code is my suggestion. Best regards, -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] LU factorization?
Charles R Harris wrote: I would just add the bits that are already there and don't add any extra dependencies, i.e., they are there when numpy is built without ATLAS or other external packages. The determinant function in linalg uses the LU decomposition, so I don't see why that shouldn't be available to the general user. If LU is already part of lapack_lite and somebody is willing to put in the work to expose the functionality to the end user in a reasonable way, then I think it should be added. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Priority rules between numpy scalars and 0d arrays
Pierre GM wrote: All, Sorry to bring back this subject, but I still haven't got any proper answers: * What are the priority rules between numpy scalars and 0d arrays ? There aren't really any specified. However, there is behavior that emerges from what is specified. The problem is that there has never been a formal resolution (that I recall) of when should something be returned as a 0-d array and when it should be returned as a scalar. There is rather an informal implementation of what actually happens. Their are some rules of thumb that have emerged (like array-operations --- e.g. reshaping --- should return 0-d arrays and not scalars). The other issue is that there is the rule that when scalars and arrays mix, the data-type of the array determines the result, but there aren't fixed rules about what the sub-type should be. The problem occurs with numpy.ma.masked, defined as a 0d, np.float64 MaskedArray, which has the __mul__ and __rmul__ of a MaskedArray. np.float(1)*ma.masked gives ma.masked, as it should np.float(64)* ma.masked gives 0, when ma.masked should have been obtained: that leads me to think that ma.masked.__rmul__ isn't called. Why ? Are 0d arrays that special beasts ? Could you post code to describe what you mean? np.float(64) should be the same type as np.float(1) so I don't get what you are saying exactly. I think the issue is that numpy scalars are currently wrapped into 0-d arrays for all math and so the 'priority' issue might really an issue between numpy arrays and masked arrays. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dtype behavior
ctw wrote:
Hi -- Can somebody here explain the following behavior:
In [1]: tst = np.array([5.])
In [2]: tst
Out[2]: array([ 5.])
In [3]: tst.shape
Out[3]: (1,)
In [4]: tst.dtype
Out[4]: dtype('float64')
In [5]: tst.dtype = np.int
In [6]: tst
Out[6]: array([ 0, 1075052544])
In [7]: tst.dtype
Out[7]: dtype('int32')
In [8]: tst.shape
Out[8]: (2,)
Setting attributes of the array always just change the information about
the array, they never change the memory the array points to.
In this case you've taken the bits that represent float64 and
re-interpreted them as int32 (that's why you know have a length 2 array).
So, you are exploring the floating-point bit-pattern on your computer.
If you want to cast to another data-type, then you need to use the
astype method:
tst = tst.astype(np.int)
-Travis
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Merged clean_math_config branch
David Cournapeau wrote: Hi there, Just to mention that I merged back my changes from the clean_math_config branch into trunk. The main point of the branch is to clean our math configuration. If this causes problems, please report it. I built and tested on mac os x, linux 32 bits and windows (both mingw32 and VS 2003). It breaks windows 64 bits ATM, but this will be fixed soon. The numscons built is broken as well, but the missing features are already backported from numpy.distutils to numscons; a new working version of numscons is about to be released. This is a really good thing and a lot of work. Thank you, David for doing all the heavy lifting. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_New bug?
Ravi wrote: Hi, PyArray_New seems to return a fortran contiguous array regardless of the requested array type. I am using numpy version 1.1.1. PyObject *obj = PyArray_New( PyArray_Type, 2, dims, /*whatever*/, NULL, NULL, 0, NPY_CARRAY, NULL ); PyObject *obj = PyArray_New( PyArray_Type, 2, dims, /*whatever*/, NULL, NULL, 0, NPY_FARRAY, NULL ); Both the above return a array who PyArray_ISFORTRAN( obj ) succeeds. I can verify this by checking bits 0 and 1 (LSB is bit 0) of PyArray_FLAGS. The C-API for PyArray_New is not what you were expecting.When data is NULL so that NumPy creates the array, then the flags argument is a toggle switch (i.e. any non-zero value for the flags argument means construct a Fortran array). Sorry for the confusion. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] choose() broadcasting, and Trac
Roman Bertle wrote: Hello, I have found something I call a bug in the numpy choose() method and wanted to report it in trac. Thanks for your report. I'm not sure why you are having trouble with Trac, but I've created a ticket for this problem. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] why type(array(1).tolist()) is int?
dmitrey wrote: let me also note that list(array((1))) returns Traceback (innermost last): File stdin, line 1, in module TypeError: iteration over a 0-d array D. This is expected. 0-d arrays are currently not iterable. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] will array(Python set) be ever implemented as cast method?
dmitrey wrote: hi all, will array(Python set) (and asarray, asfarray etc) ever be implemented as cast method? Use fromiter instead.We could special case set objects in array(...) if that is deemed desirable. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_Resize reference counting
Ravi wrote: Hi, Consider the following code: PyArrayObject *array = get_me_an_array_along_with_a_reference(); /* 1 */ PyArray_Dims *dims = get_me_some_dims(); /* 2 */ array = (PyArrayObject *)PyArray_Resize( array, dims, 1 ); /* 3 */ Is the code above valid? No. You need to do something like: temp = PyArray_Resize(array,...) Py_DECREF(array) array = temp -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help needed with the priority of 0d-arrays and np.scalars
Pierre GM wrote: Sorry to bump my own post, I know it's rude... However, I'm in dire need for some pointers: what are the priority rules when multiplying numpy scalars and 0d ndarrays ? Thanks a lot in advance ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion Hmm... I'm not 100% sure off the top of my head, but I would say that 0d arrays should determine the type coercion if any and the returned thing should be a numpy scalar. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_Resize reference counting
Ravi wrote: On Tuesday 30 September 2008 16:26:08 Travis E. Oliphant wrote: You need to do something like: temp = PyArray_Resize(array,...) Py_DECREF(array) array = temp In this case, since a new array may be returned, is there no way to synchronize with other objects holding a reference to the original array? Hold on!! I just went to the code and noticed that PyArray_Resize returns None. So, you certainly don't want to point array to it. The array does not get any reference count changes. There are three concepts here: 1) The array object 2) The memory for the array 3) The reference count on the array object PyArray_Resize doesn't change 1) or 3), it changes 2) So, PyObject *dummy; dummy = PyArray_Resize(array, ...) if (dummy == NULL) goto fail; Py_DECREF(dummy) is what you need to do. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Are there command similar as Matlab find command?
frank wang wrote: Hi, I am trying to find a command in numpy or python that perform similar function as Matlab find command. It will return the indexes of array that satisfy a condition. So far I have not found anything. There are several ways to do this, but what are you trying to do? Non-zero on the boolean array resulting from the condition is the most direct way: (a30).nonzero() where(a30) This returns a tuple of indices of length nd, where nd is the number of dimensions of a. (i.e. for 1-d case you need to extract the first element of the tuple to get the indices you want). But, if you are going to use these indices to access elements of the array, there are better ways to do that: a[a30] compress(a30, a) etc. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bug fix or behavior change?
Charles R Harris wrote:
Hi All,
Currently subtract for boolean arrays is defined in
/**begin repeat
* Arithmetic operators
*
* # OP = ||, ^, #
* #kind = add, subtract, multiply#
*/
static void
[EMAIL PROTECTED]@(char **args, intp *dimensions, intp *steps, void *func)
{
register intp i;
intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0];
char *i1=args[0], *i2=args[1], *op=args[2];
for(i=0; in; i++, i1+=is1, i2+=is2, op+=os) {
*((Bool *)op)=*((Bool *)i1) @OP@ *((Bool *)i2);
}
}
/**end repeat**/
Note that this might yield unexpected results if the boolean value is
not 0 or 1, which is possible using views. Note also that bitwise_xor
converts the boolean to 0 or 1 before using ^. I think subtract should
work the same way, but that would change current behavior. So... do I
make the change and call it a bug fix or leave it as it is?
I think it's a bugfix and so am +1 on the change.
-Travis
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Suggestion for recarray.view
Pierre GM wrote: All, I'd like to submit the following suggestion for recarray.view, so that it could accept two keywords like standard ndarrays do. As a change in records.py can potentially affect a lot of people (probably more than a change in numpy.ma), I'm not confident enough to commit it. Consider that as an attempt of having it peer-reviewed before inclusion. Cheers P. #--- def view(self, dtype=None, type=None): if dtype is None: return ndarray.view(self, type) elif type is None: try: if issubclass(dtype, ndarray): return ndarray.view(self, dtype) except TypeError: pass dtype = sb.dtype(dtype) if dtype.fields is None: return self.__array__().view(dtype) return ndarray.view(self, dtype) else: return ndarray.view(self, dtype, type) #--- This looks pretty good to me. +1 for adding it. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] patch for new mgrid / ogrid functionality
Stéfan van der Walt wrote: Hey all, David Kaplan implemented a generalised ogrid/mgrid, and would like to have some feedback. See http://projects.scipy.org/pipermail/numpy-discussion/2008-August/036691.html for his original email and attachment. If no one objects to such functionality, I shall review the patch and apply it. Generally, it is O.K. except that he changes the output from a numpy array to a list of numpy arrays. This is a small change, but it is an API change. I'm not sure why he found it necessary to do that, but it would be easier to apply the patch if it didn't change the API. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array flags - BUG?
Thomas Hrabe wrote:
Hello everyone,
I must report odd behaviour of the numpy arrays regarding the flags set for
each array object in C++.
Please have a look at the following code:
static PyObject* test(PyObject* self,PyObject* args){
int s[2];
s[0] = 1;
s[1] = 1;
char* value = (char*)PyMem_Malloc(2*sizeof(int));
PyObject* obj= PyArray_FromDimsAndData(1,s,NPY_INTLTR,(char*)value);
PyArrayObject* array = (PyArrayObject*) obj;
printf(%d\n,array-flags);
array-flags=0;
printf(%d\n,array-flags);
return obj;
}
You are using an old API and a new feature.If you want to be able to
create a Fortran array from data you need to use
PyArray_NewFromDescr
which allows setting flags.
-Travis
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Issue w/ ndarray.view
Pierre GM wrote: All, I was fixing MaskedArray.view for masked arrays with flexible type when I ran into a small pb with view. The new definition accepts 2 keywords dtype and type. I thought I could easily redefined MaskedArray.view as def view(self, dtype=None, type=None): output = ndarray.view(self, dtype=dtype, type=type) [some extra lines to view the mask if needed] return output Unfortunately, that doesn't work: if type is None, I get a ValueError that type must be a subtype of ndarray. Using output = ndarray.view(self, dtype=dtype, type=type or ndarray) doesn't work either if dtype was already a type, as it complains that type is given twice. I end up having to test whether dtype is None, type is None, build a tuple of arguments (args in one of (dtype,) or (type,) or (dtype,type) or ()) and parse that to ndarray.view(self, *args). That's a tad ugly and irritating. Shouldn't the checks on the arguments dealt with on the C side, instead of having to redo them in Python all the time ? That would be nicer. It's just a bit harder to do on the C-side where PyArg_ParseTuple is the heavy lifter and creates habits of argument acceptance that are different than on the Python side. As a side-note, recarray.view still accepts only one argument. Looks like there's a pb of consistency. Yes, it should be changed also. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dealing with types in extension modules
Lane Brooks wrote: When writing an numpy extension module, what is the preferred way to deal with the all the possible types an ndarray can have? I have some data processing functions I need to implement and they need to be generic and work for all the possible numerical dtypes. I do not want to have to re-implement the same C-code for all the possible types, so the way I approached it was to use a C++ template function to implement the processing. Then I have a dispatching function that checks the type of the input ndarray and calls the correct template. Is there a better way? You could store the functions in an array of function pointers and look-up the correct one using the typenum: resize_funcs[PyArray_Type(buf1)](PyArray_DATA(bufi)) with resize_funcs filled appropriately. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dealing with types in extension modules
Lane Brooks wrote: Travis E. Oliphant wrote: Lane Brooks wrote: When writing an numpy extension module, what is the preferred way to deal with the all the possible types an ndarray can have? I have some data processing functions I need to implement and they need to be generic and work for all the possible numerical dtypes. I do not want to have to re-implement the same C-code for all the possible types, so the way I approached it was to use a C++ template function to implement the processing. Then I have a dispatching function that checks the type of the input ndarray and calls the correct template. Is there a better way? You could store the functions in an array of function pointers and look-up the correct one using the typenum: resize_funcs[PyArray_Type(buf1)](PyArray_DATA(bufi)) with resize_funcs filled appropriately. -Travis Would this require implementing a unique function for each of the possible types, though? That is mostly what I want to avoid. I do not want to have to implement 10 to 15 different functions that all do the same exact thing but to different types of data. I guess with your proposal I can still use templates to have a single function definition. You could have a default function which does type coercion or uses type-indifferent code. It really depends on what you are doing. But generally if you want to support multiple types you have to repeat the code for that type, I don't know of anyway around that. Also, you don't have to fill in all the functions (some could be NULL and you could use coercion to another type or some other default implementation as needed). -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] NumPy arrays that use memory allocated from other libraries or tools
I wanted to point anybody interested to a blog post that describes a useful pattern for having a NumPy array that points to the memory created by a different memory manager than the standard one used by NumPy. The pattern shows how to create a NumPy array that points to previously allocated memory and then shows how to construct an object that allows the correct deallocator to be called when the NumPy array is freed. This may be useful if you are wrapping code that has it's own memory management scheme. Comments and feedback is welcome. The post is http://blog.enthought.com/?p=62 Best regards, -Travis Oliphant ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1.3: cleaning the math configuration, toward a warning free build
Should I write a nep for that too ? Or a patch is enough ? I think a patch with a useful explanation of the patch in the ticket is sufficient. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] BUG in numpy.loadtxt?
Ryan May wrote: Stefan (or anyone else who can comment), It appears that the usecols argument to loadtxt no longer accepts numpy arrays: Could you enter a ticket so we don't lose track of this. I don't remember anything being intentional. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Getting a numpy array from a ctype pointer
Paulo J. S. Silva wrote: Hello, I am writing some code interfacing C and Python using ctypes. In a callback function (in Python) I get in a parameter x which is c_double pointer and a parameter n which is c_int representing the length of the array. How can I transform this information into a numpy array? Something like this may work: from numpy import ctypeslib r = ctypeslib.as_array(x._type_ * n) If that doesn't work, then you can create an array from an arbitrary buffer or any object that exposes the __array_interface__ attribute. So, if you can get the address of c_double then you can use it as the data for the ndarray. Ask if you need more help. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Getting a numpy array from a ctype pointer
Paulo J. S. Silva wrote: After some trial and erros, I found a solution, described below. Is this the best one? Looks a little convoluted to me (C represents ctypes module and np numpy): Array = n*C.c_double x = Array.from_address(C.addressof(x.contents)) x = np.ctypeslib.as_array(x) That's a pretty simple approach. There is a faster approach which uses the undocumented function int_asbuffer function from numpy.core.multiarray (i.e. from numpy.core.multiarray import int_asbuffer) Then: x = np.frombuffer(int_asbuffer(C.addressof(x.contents), n*8)) If you don't like the hard-coded '8', then you can get that number from np.dtype(float).itemsize There may also be a way to get it from ctypes. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] rc1 update
Andrew Straw wrote: Joining the party late, here, so I don't know how much has already been said... Particularly in favor of renaming NPY_VERSION. What's the benefit? I'm -1 on removing the name NPY_VERSION. It would cause unnecessary breakage. I'm -0 on defining an additional NPY_ABI_VERSION to equal NPY_VERSION. It just seems unnecessary to me. Won't a comment do, e.g. /* We wanted to name this NPY_ABI_VERSION, but it was already NPY_VERSION since pre 1.0, so we're sticking with it. (Until 1.2, it also served a dual role as NPY_API_VERSION, when we added that, below.) */ ? I'm +0 on renaming the recently added NPY_FEATURE_VERSION to NPY_API_VERSION. I don't care what it's called, really. Since A) this is a C developer thing only (aka people who should know what they're doing), B) there is already IMO fairly good documentation immediately above the NPY_VERSION and NPY_FEATURE_VERSION in the header file, and C) very little impetus for anyone to actually directly use NPY_VERSION anyway (there's a runtime ABI check during import_array() which is the most important thing -- it checks at runtime if the compile time ABI version of the caller and numpy itself are the same), I think it's best to leave NPY_VERSION as NPY_VERSION. The only code that I know of that currently explicitly uses NPY_VERSION is C preprocessor tests for which C API to use -- from now, new code being written will check the new NPY_FEATURE_VERSION (or NPY_API_VERSION or whatever it ends up being called) for API features. Anyone updating old code will already have a bit of additional complexity to deal with, and I want to minimize the amount of additional complexity. We don't want a rat's nest of C preprossor #ifdefs as people track the numpy version checking defines. Given the addition of NPY_FEATURE_VERSION, code that's already been written with explicit NPY_VERSION checks for API information is already going to become outdated, but as long as we keep the NPY_VERSION name, at least it won't break or require another level of complexity to maintain forward compatibility. +1 on what Andrew said. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy08] Documentation BoF
Pauli Virtanen wrote: - I vaguely remember someone mentioned having done numpybook - RST conversion. If so, is the result somewhere available? Was something done towards this in the Scipy'08 sprints? Yes, Gabriel Gellner made progress on this during the sprints, and I asked him to post his scripts for doing it.I don't have his email address currently, but perhaps he will respond. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] How to replace a loop for a ufunc (to test optimizations)
Yesterday at the SciPy conference I suggested it would be a good idea to
add a function to replace one of the inner loops in a ufunc so that an
extension module could test an optimized inner loop more easily.
It turns out that this is such a good idea that it's actually already in
the code. The UFUNC_API call that does this is called:
PyUFunc_ReplaceLoopBySignature
You would use it something like this
static void
_new_add_loopfunc(char **args, intp *dimensions, intp *steps, void *func)
{
/* my new faster code */
}
Then in the extension module (either in the init or in a method call)
something like this:
PyUFuncObject *ufunc;
PyUFuncGenericFunction oldloopfunc, dummy;
int signature[3] /* This should be the number of arguments to the
ufunc you are replacing */
PyObject *module, *dict;
module = PyImport_ImportModule(umath);
dict = PyModule_GetDict(module);
ufunc = PyDict_GetItemString(dict, add) /* get the ufunc where you
want to replace one of the loops */
signature[0] = NPY_DOUBLE;
signature[1] = NPY_DOUBLE;
signature[2] = NPY_DOUBLE;
if (PyUFunc_ReplaceLoopBySignature(ufunc, _new_add_loopfunc, signature,
old1dloop) 0) {
PyErr_SetString(PyExc_RuntimeError, problem replacing loop);
/* error return i.e. return NULL or return or return -1*/
}
Then to restore the original function:
if (PyUFunc_ReplaceLoopBySignature(ufunc, old1dloop, signature, dummy)
0) {
/* error */
}
Remember to use import_umath() in the module you are using the UFUNC_API
in. Now, lot's of folks should be able to test optimizations.
-Travis
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Report from SciPy
Hi everybody, Robert K, Chuck H, Stefan VdW, Jarrod M, David C, and I had a nice discussion about the future directions of NumPy. We resolved some things and would like community feedback on them if there are opinions. * we will be moving to time-based releases (at least 2 times a year -- November / May) with major changes not accepted about 4 weeks before the release. * The releases will be numbered major.minor.bugfix * There will be no ABI changes in minor releases * There will be no API changes in bugfix releases * Any API changes in minor releases will be done in a backward compatible fashion (possibly with deprecations). * Thus 1.2 will not break ABI compatibility but will add new API features. * We will push the generalized ufuncs off to 1.3 * NumPy 2.0 will be a library and will not automagically import numpy.fft * We will suggest that other libraries use from numpy import fft instead of import numpy as np; np.fft * We will apply most of Andrew Dalke's speed up patches but will keep ctypeslib import * We will remove automatic import of numpy.doc from trunk * NumPy 1.2 will come out shortly after the conference (rc1 on Sunday). * SciPy 0.7b1 will come out after the sprints. If there is anything else I missed from those who were there, please let us all know. By the way, as promised, the NumPy book is now available for download and the source to the book is checked in to the numpy SVN tree: http://svn.scipy.org/svn/numpy/trunk/numpy/doc/numpybook Best regards, -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Generalized ufuncs?
The good news is that the patch just uses of the existing code to deal with all the tricky issues (this is why the patch is so short). By the way, sort could be implemented with the proposed specifications, its signature would be (i)-(i). I agree that it would be nice if that code could be made somewhat clearer; however, I think that this task is orthogonal to the generalized ufuncs patch, because there is no code overlap. I agree with this. The way the suggested implementation basically works is to remove the core dimensions from the input/output arrays, and then have the existing code handle all the intricacies over the loop dimensions. Reduce methods are currently not supported (an error is raised). Therefore, the current patch does not forestall anything and the desired functionality can be added whenever it is clear what would be best. I do not think that it would makes sense to specify/implement all possible extensions, optimizations, concrete ufuncs, morphing of existing numpy functions to ufuncs, etc. at once; presumably it is much better to start with a small but extremely flexible specification of generalized ufuncs first. One of the key reasons I'm enthused about the patch is because it's so small. By enhancing the ufunc object and without changing the signature of the underlying function, the patch is able to implement the general description of a generalized ufunc. I think it is useful to evaluate whether or not a few more changes will allow more functionality with little cost, but I don't think it is worth holding up the patch hoping that the code will get cleaned-up (which all code needs according to somebody's definition of cleaning). -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C-API change for 1.2
David Cournapeau wrote: On Sat, Aug 16, 2008 at 11:59 PM, David Cournapeau [EMAIL PROTECTED] wrote: On Sat, Aug 16, 2008 at 11:16 PM, Charles R Harris [EMAIL PROTECTED] wrote: I'm slowly coming to the conviction that there should be no C-ABI changes in 1.2. It does not make sense to revert those changes anymore, Actually, I did not follow the discussion when this change happened, but it does not look difficult to change the code such as we do not break the ABI. Instead of replacing the flag, we can put it at the end, and deprecate (but not remove) the old one. Would anyone be strongly against that ? No, we could do that. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C-API change for 1.2
Charles R Harris wrote: On Sat, Aug 16, 2008 at 11:21 PM, David Cournapeau [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: On Sat, Aug 16, 2008 at 11:59 PM, David Cournapeau [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: On Sat, Aug 16, 2008 at 11:16 PM, Charles R Harris [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I'm slowly coming to the conviction that there should be no C-ABI changes in 1.2. It does not make sense to revert those changes anymore, Actually, I did not follow the discussion when this change happened, but it does not look difficult to change the code such as we do not break the ABI. Instead of replacing the flag, we can put it at the end, and deprecate (but not remove) the old one. Would anyone be strongly against that ? I have nothing against extensions when they can be made to serve. If a dictionary gets added to ndarrays I hope it is done that way, likewise for generalized ufuncs. In the present case I think Travis wants to preserve the functionality while changing the name and type, and that doesn't really fit the extension model. But I might be wrong about that. The problem was that I didn't break ABI compatibility at 1.0. I new the char was too small to hold what the field had really become (a flag field).I had intended for 1.1 to be a potential ABI breakage, but this was changed when the release strategy changed. But, there is no real functionality added by changing the ABI at this point. I'm just looking to the future, but I can be convinced that it's too late. What about the version number changes. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy 1.2.0b2 released
Andrew Dalke wrote: Andrew Dalke: Any chance of someone reviewing my suggestions for making the import somewhat faster still? http://scipy.org/scipy/numpy/ticket/874 Travis E. Oliphant: In sum: I think 2, 3, 6, 7, 8, and 9 can be done immediately. 1) and 4) could be O.K. but 1) does break code and 4 I'm not sure about.5 seems like it's too much code duplication for too little savings for my taste. Since no one else has said yea or nay, and 2.1 release draws nigh[*], the simplest solution is to do 2, 3, 6, 7, 8, and 9. I showed that 1 will break existing code. As for #4 - as far as I can tell the code in 'doc' is recent, so no user code depends on it. Plus, the documentation that's there is effectively unusable, with files like: I say go ahead including changing #1 and #4. Let's leave 5 for the moment. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy 1.2.0b2 released
Stéfan van der Walt wrote: 2008/8/18 Travis E. Oliphant [EMAIL PROTECTED]: I say go ahead including changing #1 and #4. Let's leave 5 for the moment. I ran several benchmarks and made sure that these imports take a minimal amount of time. Wouldn't we want users to have access with the doc framework without doing anything special? And, yes, some of the documents are empty, but a number of them have already been written. I still think we are going about this the wrong way. We have two different sets of expectations, and we can't satisfy both by ripping everything apart. I'd much prefer two entry points into NumPy: one for people who need speed, and one for those who need the convenience of everything being at hand. I think you are right Stefan. It would be great to have another name-space that is lighter from which numpy imports. But there is no reason to hold up these useful speed increases waiting for a better solution. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C-API change for 1.2
Jon Wright wrote: Travis, Stéfan, I missed Travis mail previously. Are you *really* sure you want force all C code which uses numpy arrays to be recompiled? Re-compilation is necessary at some point. We have not required recompilation for a long time now.Yes, it is a pain for distribution, but those who don't want to re-compile can point people to 1.1.1 which will still work until they make a new release compiled against the newer NumPy. I would encourage people to make use of things like Python(x,y), EPD, and SAGE for their distribution needs. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] long double woes on win32
Charles R Harris wrote: Yes. I don't think MS will support true long doubles any time soon and this affects printing and the math functions. I'm not sure what the best solution is, there are various possibilities. 1) We could define the numpy longdouble type to be double, which makes us compatible with MS and is effectively what numpy compiled with MSVC does since MSVC long doubles are doubles. Perhaps this could be done by adding a -DNOLONGDOUBLE flag to the compiler flags and then defining longdouble to double. But this means auditing the code to make sure the long double type is never explicitly used so that the mingw compiler does the right thing. I don't think this should be a problem otherwise except for the loss of true long doubles and their extra bit of precision. All code in numpy uses npy_longdouble which is typedef'd to double if SIZEOF_LONGDOUBLE is the same as SIZEOF_DOUBLE. I don't understand why it's a problem to just define SIZEOF_LONGDOUBLE = 8 for mingw which is what it is for MSVC.This makes longdouble 8 bytes. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Generalized ufuncs?
Numpy 1.2 is for documentation, bug fixes, and getting the new testing framework in place. Discipline is called for if we are going to have timely releases. We also agreed to a change in the C-API (or at least did not object too loudly). I'm in favor of minimizing that sort of change. Why not wait until after the release then? The biggest reason is that the patch requires changing the C-API and we are already doing that for 1.2. I would rather not do it again for another 6 months at least. I don't think we should make the patch wait that long. Your code review is very much appreciated. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Generalized ufuncs?
Can we fix the ticket notification mailings some day? It's been almost four months now. That would be fabulous. So far nobody has figured out how... Jarrod?? Re: the patch. I noticed the replacement of the signed type int by an unsigned size_t. Where did you notice this? I didn't see it. python or numpy types. The use of inline and the local declaration of variables would also have been caught early in a code review. What do you mean by the local declaration of variables? -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Generalized ufuncs?
Travis E. Oliphant wrote: Can we fix the ticket notification mailings some day? It's been almost four months now. That would be fabulous. So far nobody has figured out how... Jarrod?? Re: the patch. I noticed the replacement of the signed type int by an unsigned size_t. Where did you notice this? I didn't see it. Are you referring to Stefan's patch to the Fu's _parse_signature code in r5654.This is a local function, I'm not sure why there is a concern. python or numpy types. The use of inline and the local declaration of variables would also have been caught early in a code review. What do you mean by the local declaration of variables? Never mind, I understand it's the mid-code declaration of variables (without a separate block defined) that Stefan fixed. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy 1.2.0b2 released
Andrew Dalke wrote: I forgot to mention.. On Aug 15, 2008, at 9:00 AM, Travis E. Oliphant wrote: 1) Removing ctypeslib import * Can break code if somebody has been doing import numpy and then using numpy.ctypeslib * I'm fine with people needing to import numpy.ctypeslib to use the capability as long as we clearly indicate these breakages. You were the one who had numpy/__init__.py always import ctypeslib r3027 | oliphant | 2006-08-15 11:53:49 +0200 (Tue, 15 Aug 2006) | 1 line import ctypeslib on numpy load and change name from ctypes_load_library to load_library Was there a driving reason for that other than decreased user burden? Not that I can recall. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Addition of a dict object to all NumPy objects
Hello all, While we are on the subject of C-API changes, I've noticed that quite a few of the sub-classes of ndarray are constructed to basically add meta-information to the array. What if the base-class ndarray grew a dict object at it's end to hold meta information. Naturally, several questions arise: 1) How would you access the dictionary? (i.e. __dict__?) 2) Would attribute setting and getting retrieve from this dictionary (how are conflicts managed). * I think I would prefer a dict attribute on the numpy array that gets and sets into the dictionary. 3) Are the additional 4-8 bytes too expensive 4) Should there instead be a C-level array sub-class with the dict attached. 5) How should dict's be propagated through views? My preference is to not propagate them at all. Thoughts? -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Addition of a dict object to all NumPy objects
Robert Kern wrote: 3) Are the additional 4-8 bytes too expensive One of the problems with numarray was the time taken to allocate small arrays. Would adding a dictionary slow down the allocation of numpy arrays? No, I don't think so, not if we did nothing by default but set the dict to NULL (i.e. no propagation of meta-information to new arrays). That said, I think we should keep things as simple and orthogonal as possible. If we go this way, I think a subclass with a dictionary would be the best approach to avoid the heartbreak of creeping featuritis. The point of the feature is to avoid subclasses. There are a number of use cases for annotating arrays with metadata. Currently, they are forced to use subclasses. Every time you use ndarray subclasses, you are essentially forcing yourself into your subclass's ghetto of functions that only work on your subclass. This would be one-step better in the sense that there would be a single sub-class to handle all cases of just needing meta information.But, I tend to agree that adding the dictionary to all arrays is preferable. I think you could make the dictionary created lazily on the first getattr(). Yes, that could easily be done. It would just be set to NULL on creation and the penalty/overhead would be the extra pointer in the array structure. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy 1.2.0b2 released
Andrew Dalke wrote: Can that be made quicker? Not easily. lib is first imported in add_newdocs. Personally, I want to get rid of add_newdocs and move the docstrings into the correct locations. Where would that be, in the C-code? The reason for add_newdocs is to avoid writing docstrings in C-code which is a pain. It's possible to clean up the code so this loop doesn't exist, and fix things so that fewer things are imported when some environment variable is set, but it doesn't look easy. Modules depend on other modules a bit too much to make me happy. I've removed this loop. Are there other places in numpy.core that depend on numpy.lib? Thanks for the very helpful analysis. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] min() of array containing NaN
Thomas J. Duck wrote: Determining the minimum value of an array that contains NaN produces a surprising result: x = numpy.array([0,1,2,numpy.nan,4,5,6]) x.min() 4.0 I expected 0.0. Is this the intended behaviour or a bug? I am using numpy 1.1.1. NAN's don't play well with comparisons because comparison with them is undefined.See numpy.nanmin -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Appending data to a big ndarray
oc-spam66 wrote: Hello, I would like to build a big ndarray by adding rows progressively. I considered the following functions : append, concatenate, vstack and the like. It appears to me that they all create a new array (which requires twice the memory). Is there a method for just adding a row to a ndarray without duplicating the data ? You can resize the memory for an array with the resize method.You have to be careful using this approach if you have other views looking at the same memory because the resize method may move the pointer which is very bad for all the other items looking at it: a = array([1,2,3,4]) a.resize((10,4)) -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] C-API change for 1.2
Hi all, The 1.2 version of NumPy is going to be tagged. There is at least one change I'd like to add: The hasobject member of the PyArray_Descr structure should be renamed to flags and converted to a 32-bit integer. What does everybody think about this change? It should have minimal affect except to require a re-compile of extension modules using NumPy. The only people requiring code changes would be those making intimate use of the PyArray_Descr structure instead of using the macros. It's a simple change if there is no major opposition. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Error creating an array
Sameer DCosta wrote: Im having a little trouble creating a numpy array with a specific dtype. I can create the array b with dtype=int, but not with the dtype I want. Any idea what I am doing wrong here? You must uses tuples for the individual records when constructing arrays with the array command. Thus, data = [(1,2), (3,4), (5,6)] will work. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] No Copy Reduce Operations
Luis Pedro Coelho wrote: Hello all, Numpy arrays come with several reduce operations: sum(), std(), argmin(), min(), The traditional implementation of these suffers from two big problems: It is slow and it often allocates intermediate memory. I have code that is failing with OOM (out of memory) exceptions in calls to ndarray.std(). I regularly handle arrays with 100 million entries (have a couple of million objects * 20 features per object = 100 million doubles), so this is a real problem for me. Luis, Thank you for your work and your enthusiasm. You are absolutely right that the default implementations are generic and therefore potentially slower and more memory consuming. There is generally a basic tradeoff between generic code and fast/memory-conserving code. The default implementations of std and var are much different than sum, min, argmin, etc. The main difference is that the latter are direct reduce methods on the ufuncs while the former are generic extensions using python written with the Python C-API. Your approach using C++ templates is interesting, and I'm very glad for your explanation and your releasing of the code as open source.I'm not prepared to start using C++ in NumPy, however, so your code will have to serve as an example only. One way to do this without using templates is to extend the dtype functions array with additional function pointers (std, and var). This has been done several times in the past and it is probably advisable. In that case your code could very likely be used (using the C-compatible portions). I'm grateful you are willing to re-license that part of your code as BSD so that it can possibly be used in NumPy. Thanks so much. It is exciting to see interest in the code especially from students. Best regards, -Travis P.S. Specific answers to some of your questions below. I am not correctly implementing the dtype parameter. I thought it controlled the type of the intermediate results, but if I do The dtype parameter controls the type of the reduction but not the final result (which is always a float for the mean because of the division). I see three possible paths for this: (1) You agree that this is nice, it achieves good results and I should strive to integrate this into numpy proper, replacing the current implementation. I would start a branch in numpy svn and implement it there and finally merge into the main branch. I would be perfectly happy to relicense to BSD if this is the case. The idea of your code is great, but the C++ implementation cannot be directly used. One could think of the array-to-array (A+B, A+=2, A != B,...) operations using a similar scheme This would imply that part of the C API would be obsoleted: the whole functions array would stop making sense. I don't know how widespread it's used. I don't know what you mean by the functions array. If you are talking about the ufuncs, then yes it is widespread. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy fails with multiarray.so: undefined symbol: PyUnicodeUCS2_FromUnicode
G wrote: Hello, I have installed the svn version of numpy. I have deleted all previous versions of and files related to numpy prior to installing. I also have tried reinstalling python from source. Regardless, when I try import numpy, I get the following: Python 2.5.2 (r252:60911, Jul 23 2008, 23:54:29) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2 Type help, copyright, credits or license for more information. import numpy Traceback (most recent call last): File stdin, line 1, in module File /usr/local/lib/python2.5/site-packages/numpy/__init__.py, line 93, in module import add_newdocs File /usr/local/lib/python2.5/site-packages/numpy/add_newdocs.py, line 9, in module from lib import add_newdoc File /usr/local/lib/python2.5/site-packages/numpy/lib/__init__.py, line 4, in module from type_check import * File /usr/local/lib/python2.5/site-packages/numpy/lib/type_check.py, line 8, in module import numpy.core.numeric as _nx File /usr/local/lib/python2.5/site-packages/numpy/core/__init__.py, line 5, in module import multiarray ImportError: /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_FromUnicode This symbol is defined by Python when you build Python with 16-bit unicode support (as opposed to 32-bit) unicode support. Somehow numpy is picking up the 16-bit headers but you probably compiled with ucs4. NumPy supports both UCS2 and UCS4 builds. This looks to me like a Python header installation problem.There are probably some incorrect Python headers being picked up during compilation of NumPy. Can you double check which Python headers are being used (look at the -I lines when NumPy is being built). -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ticket #842.
Stéfan van der Walt wrote: 2008/7/19 Charles R Harris [EMAIL PROTECTED]: In [2]: type(conjugate(array(8+7j))) Out[2]: type 'numpy.complex128' In [3]: type((array(8+7j))) Out[3]: type 'numpy.ndarray' So I think all that needs to be done is fix the return type conjugate if we agree that it should be an array. I think it should be an array. +1/2 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket review: #848, leak in PyArray_DescrFromType
Michael Abbott wrote: I'm not actually convinced by the comment that's there now, which says /* typecode will be NULL */ but in truth it doesn't matter -- because of the correcly placed DECREF after the PyArray_Scalar calls the routine no longer owns typecode. I'm pretty sure that it's fine. If I can refer to my last message, I made the point that there wasn't a good invariant at the finish label -- we didn't know how many references to typecode we were responsible for at that point -- and I offered the solution to keep typecode. Instead you have chosen to recreate typecode, which I hadn't realised was just as good. I agree that this routine needs aesthetic improvement. I had hoped that someone would have improved the array scalars routines by now. I think a core issue is that to save a couple of lines of code, an inappropriate goto finish in the macro was used. This complicated the code more than the savings of a couple lines of code would justify. Really this code grew from a simple thing into a complicated thing as more features were added. This is a common issue that happens all over the place. This code is still horrible, but I don't think I want to try to understand it anymore. It'd be really nice (it'd make me feel a lot better) if you'd agree that my original patch was in fact correct. I'm not disputing the correcness of the current solution (except I think that typecode can end up being created twice, but who really cares?) but I've put a lot of effort into arguing my case, and the fact is my original patch was not wrong. From what I saw, I'm still not quite sure. Your description of reference counting was correct and it is clear you've studied the issue which is great, because there aren't that many people who understand reference counting on the C-level in Python anymore and it is still a useful skill. I'm hopeful that your description of reference counting will be something others can find and learn from. The reason I say I'm not sure is that I don't remember seeing a DECREF after the PyArray_Scalar in the obj = NULL part of the code in my looking at your patches.But, I could have just missed it. Regardless, that core piece was lost in my trying to figure out the other changes you were making to the code. From a generic reference counting point of view you did correctly emphasize the problem of having a reference count creation occur in an if-statement but a DECREF occur all the time in the finish: section of the code. It was really that statement: the fantasy that PyArray_Scalar steals a reference that tipped me off to what I consider one of the real problems to be.The fact that it was masked at the end of a long discussion about other reference counting and a stealing discussion that were not the core problem was distracting and ultimately not very helpful. I'm very impressed with your ability to follow these reference count issues. Especially given that you only started learning about the Python C-API a few months ago (if I remember correctly).I'm also very glad you are checking these corner cases which have not received the testing that they deserve.I hope we have not discouraged you too much from continuing to help. Your input is highly valued. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Branch cuts, inf, nan, C99 compliance
Pauli Virtanen wrote: Hi all, Re: Ticket 854. I wrote tests for the branch cuts for all complex arc* functions in umathmodule. It turns out that all except arccosh were OK. The formula for arcsinh was written in a non-standard form with an unnecessary nc_neg, but this didn't affect the results. I also wrote tests for checking values of the functions at infs and nans. Thanks for looking into these. These functions were contributed by Konrad Hinsen (IIRC) many years ago and I don't think they've really been reviewed since then. I'm all for using C99 when it is available and improving these functions with help from cmathmodule. IIRC, the cmathmodule was contributed by Konrad originally also. So +1 on C99 standardization. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Another reference count leak: ticket #848
Looking at the uses of PyArray_FromAny I can see the motivation for this design: core/include/numpy/ndarrayobject.h has a lot of calls which take a value returned by PyArray_DescrFromType as argument. This has prompted me to take a trawl through the code to see what else is going on, and I note a couple more issues with patches below. The core issue is that NumPy grew out of Numeric. In Numeric PyArray_Descr was just a C-structure, but in NumPy it is now a real Python object with reference counting. Trying to have a compatible C-API to the old one and making the transition with out huge changes to the C-API is what led to the stealing strategy. I did not just out the blue decide to do it that way. Yes, it is a bit of a pain, and yes, it isn't the way it is always done, but as you point out there are precedents, and so that's the direction I took. It is *way* too late to change that in any meaningful way In the patch below the problem being fixed is that the first call to PyArray_FromAny can result in the erasure of dtype *before* Py_INCREF is called. Perhaps you can argue that this only occurs when NULL is returned... Indeed I would argue that because the array object holds a reference to the typecode (data-type object). Only if the call returns NULL does the data-type object lose it's reference count, and in fact that works out rather nicely and avoids a host of extra Py_DECREFs. The next patch deals with an interestingly subtle memory leak in _string_richcompare where if casting to a common type fails then a reference count will leaked. Actually this one has nothing to do with PyArray_FromAny, but I spotted it in passing. This is a good catch. Thanks! I really don't think that this design of reference count handling in PyArray_FromAny (and consequently PyArray_CheckFromAny) is a good idea. Your point is well noted, but again given the provenance of the code, I still think it was the best choice. And, yes, it is too late to change it. Not only is this not a good idea, it's not documented in the API documentation (I'm referring to the Guide to NumPy book) Hmmm... Are you sure? I remember writing a bit about in the paragraphs that describe the relevant API calls. But, you could be right. I've been trying to find some documentation on stealing references. The Python C API reference (http://docs.python.org/api/refcountDetails.html) says Few functions steal references; the two notable exceptions are PyList_SetItem() and PyTuple_SetItem() An interesting essay on reference counting is at http://lists.blender.org/pipermail/bf-python/2005-September/003092.html Believe me, I understand reference counting pretty well. Still, it can be tricky to do correctly and it is easy to forget corner cases and error-returns. I very much appreciate your careful analysis, but I did an analysis of my own when I wrote the code, and so I will be resistant to change things if I can't see the error. I read something from Guido once that said something to the effect that nothing beats studying the code to get reference counting right. I think this is true. In conclusion, I can't find much about the role of stealing in reference count management, but it's such a source of surprise (and frankly doesn't actually work out all that well in numpy) I strongly beg to differ. This sounds very naive to me.IMHO it has worked out extremely well in converting the PyArray_Descr C-structure into the data-type objects that adds so much power to NumPy. Yes, there are a few corner cases that you have done an excellent job in digging up, but they are corner cases that don't cause problems for 99.9% of the use-cases. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket review: #848, leak in PyArray_DescrFromType
Michael Abbott wrote: Only half of my patch for this bug has gone into trunk, and without the rest of my patch there remains a leak. Thanks for your work Michael. I've been so grateful to have you and Chuck and others looking carefully at the code to fix its problems. In this particular case, I'm not sure I see how (the rest of) your patch fixes any remaining leak. We do seem to be having a disagreement about whether or not the reference to typecode can be pre-maturely destroyed, but this doesn't fit what I usually call a memory leak. I think there may be some other cause for remaining leaks. I can see that there might be an argument that PyArray_FromAny has the semantics that it retains a reference to typecode unless it returns NULL ... but I don't want to go there. That would not be a good thing to rely on -- and even with those semantics the existing code still needs fixing. Good, that is the argument I'm making. Why don't you want to rely on it? Thanks for all your help. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket review: #848, leak in PyArray_DescrFromType
Charles R Harris wrote: The reference leak seems specific to the float32 and complex64 types called with default arguments. In [1]: import sys, gc In [2]: t = float32 In [3]: sys.getrefcount(dtype(t)) Out[3]: 4 In [4]: for i in range(10) : t(); ...: In [5]: sys.getrefcount(dtype(t)) Out[5]: 14 In [6]: for i in range(10) : t(0); ...: In [7]: sys.getrefcount(dtype(t)) Out[7]: 14 In [8]: t = complex64 In [9]: sys.getrefcount(dtype(t)) Out[9]: 4 In [10]: for i in range(10) : t(); : In [11]: sys.getrefcount(dtype(t)) Out[11]: 14 In [12]: t = float64 In [13]: sys.getrefcount(dtype(t)) Out[13]: 19 In [14]: for i in range(10) : t(); : In [15]: sys.getrefcount(dtype(t)) Out[15]: 19 This shouldn't actually leak any memory as these types are singletons, but it points up a logic flaw somewhere. That is correct. There is no memory leak, but we do need to get it right. I appreciate the extra eyes on some of these intimate details. What can happen (after a lot of calls) is that the reference count can wrap around to 0 and then cause a funny dealloc (actually, the dealloc won't happen, but a warning will be printed). Fixing the reference counting issues has been the single biggest difficulty in converting PyArray_Descr from a C-structure to a regular Python object. It was a very large pain to begin with, and then there has been more code added since the original conversion some of which does not do reference counting correctly (mostly my fault). I've looked over ticket #848 quite a bit and would like to determine the true cause of the growing reference count which I don't believe is fixed by the rest of the patch that is submitted there. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket review: #848, leak in PyArray_DescrFromType
Charles R Harris wrote: On Fri, Jul 18, 2008 at 9:15 PM, Travis E. Oliphant [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Michael Abbott wrote: Only half of my patch for this bug has gone into trunk, and without the rest of my patch there remains a leak. Thanks for your work Michael. I've been so grateful to have you and Chuck and others looking carefully at the code to fix its problems. In this particular case, I'm not sure I see how (the rest of) your patch fixes any remaining leak. We do seem to be having a disagreement about whether or not the reference to typecode can be pre-maturely destroyed, but this doesn't fit what I usually call a memory leak. I think there may be some other cause for remaining leaks. Travis, There really is (at least) one reference counting error in PyArray_FromAny. In particular, the obj == NULL case leaves a reference to typecode, then exits through the first return after finish. In this case robj doesn't steal a reference to typecode and the result can be seen in the python program above or by printing out the typecode-ob_refcnt from the code itself. So that needs fixing. I would suggest a DECREF in that section and a direct return of robj. The next section before finish is also a bit odd. The direct return of an array works fine, but if that isn't the branch taken, then PyArray_Return decrements the refcnt of arr, which in turn decrements the refcnt of typecode. I don't know if the resulting scalar holds a reference to typecode, but in any case the situation there should also be clarified. Thank you. I will direct attention there and try to clear this up tonight. I know Michael is finding problems that do need fixing. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket review: #848, leak in PyArray_DescrFromType
Charles R Harris wrote: On Fri, Jul 18, 2008 at 9:15 PM, Travis E. Oliphant [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Michael Abbott wrote: Only half of my patch for this bug has gone into trunk, and without the rest of my patch there remains a leak. Thanks for your work Michael. I've been so grateful to have you and Chuck and others looking carefully at the code to fix its problems. In this particular case, I'm not sure I see how (the rest of) your patch fixes any remaining leak. We do seem to be having a disagreement about whether or not the reference to typecode can be pre-maturely destroyed, but this doesn't fit what I usually call a memory leak. I think there may be some other cause for remaining leaks. Travis, There really is (at least) one reference counting error in PyArray_FromAny. In particular, the obj == NULL case leaves a reference to typecode, then exits through the first return after finish. In this case robj doesn't steal a reference to typecode and the result can be seen in the python program above or by printing out the typecode-ob_refcnt from the code itself. So that needs fixing. I would suggest a DECREF in that section and a direct return of robj. agreed! I'll commit the change. The next section before finish is also a bit odd. The direct return of an array works fine, but if that isn't the branch taken, then PyArray_Return decrements the refcnt of arr, which in turn decrements the refcnt of typecode. I don't know if the resulting scalar holds a reference to typecode, but in any case the situation there should also be clarified. This looks fine to me. At the PyArray_Return call, the typecode reference is held by the array. When it is decref'd the typecode is decref'd appropriately as well. The resulting scalar does *not* contain a reference to typecode. The scalar C-structure has no place to put it (it's just a PyObject_HEAD and the memory for the scalar value). Michael is correct that PyArray_Scalar does not change the reference count of typecode (as the comments above that function indicates). I tried to be careful to put comments near the functions that deal with PyArray_Descr objects to describe how they affect reference counting. I also thought I put that in my book. -Travis -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket review: #848, leak in PyArray_DescrFromType
Charles R Harris wrote: On Fri, Jul 18, 2008 at 9:15 PM, Travis E. Oliphant [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Michael Abbott wrote: Only half of my patch for this bug has gone into trunk, and without the rest of my patch there remains a leak. Thanks for your work Michael. I've been so grateful to have you and Chuck and others looking carefully at the code to fix its problems. In this particular case, I'm not sure I see how (the rest of) your patch fixes any remaining leak. We do seem to be having a disagreement about whether or not the reference to typecode can be pre-maturely destroyed, but this doesn't fit what I usually call a memory leak. I think there may be some other cause for remaining leaks. Travis, There really is (at least) one reference counting error in PyArray_FromAny. In particular, the obj == NULL case leaves a reference to typecode, then exits through the first return after finish. In this case robj doesn't steal a reference to typecode and the result can be seen in the python program above or by printing out the typecode-ob_refcnt from the code itself. So that needs fixing. I would suggest a DECREF in that section and a direct return of robj. agreed! I'll commit the change. The next section before finish is also a bit odd. The direct return of an array works fine, but if that isn't the branch taken, then PyArray_Return decrements the refcnt of arr, which in turn decrements the refcnt of typecode. I don't know if the resulting scalar holds a reference to typecode, but in any case the situation there should also be clarified. This looks fine to me. At the PyArray_Return call, the typecode reference is held by the array. When it is decref'd the typecode is decref'd appropriately as well. The resulting scalar does *not* contain a reference to typecode. The scalar C-structure has no place to put it (it's just a PyObject_HEAD and the memory for the scalar value). Michael is correct that PyArray_Scalar does not change the reference count of typecode (as the comments above that function indicates). I tried to be careful to put comments near the functions that deal with PyArray_Descr objects to describe how they affect reference counting. I also thought I put that in my book. -Travis -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket review: #848, leak in PyArray_DescrFromType
I've attached a test script. Thank you! It looks like with that added DECREF, the reference count leak is gone.While it was a minor issue (it should be noted that reference counting errors on the built-in data-types won't cause issues), it is nice to clean these things up when we can. I agree that the arrtype_new function is hairy, and I apologize for that. The scalartypes.inc.src was written very quickly. I added a few more comments in the change to the function (and removed a hard-coded hackish multiply with one that takes into account the actual size of Py_UNICODE). -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] snprintf vs PyOS_snprintf
Robert Kern wrote: On Sat, Jul 12, 2008 at 13:11, Charles R Harris [EMAIL PROTECTED] wrote: Numpy uses a mix of snprintf and PyOS_snprintf. The Python version is there because snprintf wasn't part of the standard until C99. So either we should stick to the python version or make the decision that we only support compilers with a working snprintf. Which way should we go? PyOS_snprintf +1 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Another reference count leak: ticket #848
Michael Abbott wrote: The attached patch fixes another reference count leak in the use of PyArray_DescrFromType. The first part of this patch is good. The second is not needed. Also, it would be useful if you could write a test case that shows what is leaking and how you determined that it is leaking. Could I ask that both this patch and my earlier one (ticket #843) be applied to subversion. Thank you. Definitely not enjoying this low level code. What doesn't kill you makes you stronger :-) But, you are correct that reference counting is a bear. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] chararray behavior
Alan McIntyre wrote: Since chararray doesn't currently have any tests, I'm writing some, and I ran across a couple of things that didn't make sense to me: 1. The code for __mul__ is exactly the same as that for __rmul__; is there any reason __rmul__ shouldn't just call __mul__? Just additional function call overhead, but it's probably fine to just call __mul__. 1.5. __radd__ seems like it doesn't do anything fundamentally different from __add__, is there a reason to have a separate implementation of __radd__? Possibly. I'm not sure. 2. The behavior of __mul__ seems odd: What is odd about this? It is patterned after 'a' * 3 'a' * 4 'a' * 5 for regular python strings. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Another reference count leak: ticket #848
Michael Abbott wrote: On Tue, 8 Jul 2008, Travis E. Oliphant wrote: Michael Abbott wrote: The attached patch fixes another reference count leak in the use of PyArray_DescrFromType. The first part of this patch is good. The second is not needed. I don't see that. The second part of the patch addresses the case of an early return: this means that the DECREF that occurs later on in the code is bypassed, and so a reference leak will still occur if this early return case occurs. Don't forget that PyArray_DescrFromType returns an incremented reference that has to be decremented, returned or explicitly assigned -- the DECREF obligation has to be met somewhere. Don't forget that PyArray_FromAny consumes the reference even if it returns with an error. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy, buffers (PEP 3118), complex floats, and Cython
Dag Sverre Seljebotn wrote:
I'd like some advice for what way people feel would be the best for
supporting complex datatypes in NumPy in Cython; as well as ask in what
way it is likely that NumPy will make use of PEP 3118.
It seems like NumPy defines its complex data to be a struct of two
doubles, for instance:
typedef struct { double real, imag; } npy_cdouble;
According to PEP 3118 [1], it would be natural to export this as dd
(record of two doubles) rather than Zd (complex double), when
exporting ndarrays through a buffer interface. Right?
No, it is more natural to use Zd because then you know you are dealing
with complex numbers and not just two separate floats.
For Cython, it seems most user-friendly to use the C99 standard for
complex numbers (not currently supported, but it wouldn't be much work)
and have code like this:
cdef ndarray[complex double, 2] arr = ...
cdef complex double item
item = arr[34, 23]
That is fine to use the C99 standard.
So the natural questions then are:
- Is it ok to assume that (complex double*) can be safely casted to
(npy_cdouble*) on all platforms which both a) NumPy compiles and runs
on, and b) supports complex double? (It seems to be ok with GCC but I
wouldn't know about other compilers.)
I think so, but I'm not 100% sure.
- Will NumPy ever be rewritten to use the C99 complex datatype if
available? (I.e. #ifdef how npy_cdouble is defined, and define
corresponding #ifdef-ed macros for all complex operations). This would
remove the need for such a cast.
Can't answer that one. C99 is not supported by Microsoft's compilers
and other compilers and so could not be used though I would have liked to.
- Does NumPy plan to support PEP3118, and if so, will complex numbers be
exported as dd or Zd?
Yes, it will support it eventually. PEP3118 came from NumPy. Complex
numbers will be exported as Zd
-Travis
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Time to fix ticket #390?
Charles R Harris wrote: Questions about ticket #390 http://projects.scipy.org/scipy/numpy/ticket/390 : 1. Should we fix this for 1.2? +1 2. Is the attached patch the correct fix for this? It looks good, except for the rare case where the array contains the word rec Perhaps the fix should be changed to only replace the rec only at the first of the string to rec. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Should the fields in void scalars be available?
Robert Kern wrote: The only inconsistency I can see here is that x.item() gives a tuple rather than a scalar record. A scalar record does have field access, but the tuple doesn't. In [28]: x.item()['a'] .item() always returns a standard Python type. So, this is expected behavior. If you want to index a 0-d array to get the array scalar do x[()] or x[...] -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Review of issue 825
Charles R Harris wrote:
On Wed, Jun 25, 2008 at 2:04 PM, Neil Muller
[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote:
On Wed, Jun 25, 2008 at 7:53 PM, Charles R Harris
[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote:
But I wonder if this case isn't supposed to be caught by this
fragment:
if (!PyArray_ISBEHAVED(ap)) {
buffer = _pya_malloc(mysize 2);
if (buffer == NULL)
return PyErr_NoMemory();
alloc = 1;
memcpy(buffer, ip, mysize 2);
if (!PyArray_ISNOTSWAPPED(ap)) {
byte_swap_vector(buffer, mysize, 4);
}
}
This actually works fine - the problem is immediately before this,
with the loop to find the end of the unicode string.
This is indeed the problem. No de-referencing of anything but a char
pointer should be done unless you are sure you have an aligned array.
This length-of-string adjustment code should definitely be placed after
the copy is done so that an aligned array is obtained.
The other approach is to check the length of string differently (i.e.
using only byte dereferencing) so that when the string is misaligned
un-necessary copies are not done. But, Chuck's patch looks sufficient.
Best regards,
-Travis
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Request for clarification from Travis
Yes, but it is directed at the interpreter, which will raise a TypeError if needed. But the interpreter doesn't know that some generic function might return NotImplemented and wouldn't know what to do if it did. What the user should see when they call something like right_shift(a,b) and it fails is an exception, they shouldn't have to check the return value Yes, you are right. So, the questions is how to handle this correctly. I think what you want is that subtypes of ndarray can override some ufuncs so that, for instance, right_shift() should interpret itself as a call to arg2.__rrshift__ if it exists, even if it isn't called through the interpreter friendly slot function . Is that correct? If so, I would rather have an explicit overload flag in the subtype so that the ufunc can easily do the right thing. So, how would this overload flag work and how would a subtype set it? Would this be a per-instance flag in the FLAGS field? I can't find __array_priority__ in your book, what exactly does it imply? The subtype with the largest priority wins in any decision about which subtype to create during operations involving more than one subtype (like adding a matrix to an array --- what should be returned?). -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Record arrays
Stéfan van der Walt wrote: Hi all, I am documenting `recarray`, and have a question: Is its use still recommended, or has it been superseded by fancy data-types? I rarely recommend it's use (but some people do like attribute access to the fields).It is wrong, however, to say that recarray has been superseded by fancy data types because fancy data types have existed for as long as recarrays. I believe pyfits uses them quite a bit, and so they deserve to be documented. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Request for clarification from Travis
So it has a numeric value? Yes, it's just a floating point number. It's not a very elegant thing, but it does allow some ability to specify an ordering. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
