>> >> Bingo. Yes, definitely allocating new _types_ (an awful lot of >> them...) >> --- that's what the "array scalars" are: new types created in C. > > > Do you really mean that someArray[1] will create a new type to represent > the second element of someArray? I would guess that you create an > instance of a type defined in your extension.
O.K. my bad. I can see that I was confusing in my recent description and possibly misunderstood the questions I was asked. It can get confusing given the dynamic nature of Python. The array scalars are new statically defined (in C) types (just like regular Python integers and regular Python floats). The ndarray is also a statically defined type. The ndarray holds raw memory interpreted in a certain fashion (very similar to Python's array module). Each ndarray can have a certain data type. For every data type that an array can be, there is a corresponding "array scalar" type. All of these are statically defined types. We are only talking about instances of these defined types. When the result of a user operation with an ndarray is a scalar, an instance of the appropriate "array scalar" type is created and passed back to the user. Previously we were using PyObject_New in the tp_alloc slot and PyObject_Del in the tp_free slot of the typeobject structure in order to create and destroy the memory for these instances. In this particular application, the user ended up creating many, many instances of these array scalars and then deleting them soon after. Despite the fact that he was not retaining any references to these scalars (PyObject_Del had been called on them), his application crawled to a halt after only several hunderd iterations consuming all of the available system memory. To verify that indeed no references were being kept, I did a detailed analysis of the result of sys.getobjects() using a debug build of Python. When I replaced PyObject_New (with malloc and PyObject_Init) and PyObject_Del (with free) for the "array scalars" types in scipy core, the users memory problems magically disappeared. I therefore assume that the problem is the memory manager in Python. Initially, I thought this was the old problem of Python not freeing memory once it grabs it. But, that should not have been a problem here, because the code quickly frees most of the objects it creates and so Python should have been able to re-use the memory. So, I now believe that his code (plus the array scalar extension type) was actually exposing a real bug in the memory manager itself. In theory, the Python memory manager should have been able to re-use the memory for the array-scalar instances because they are always the same size. In practice, the memory was apparently not being re-used but instead new blocks were being allocated to handle the load. His code is quite complicated and it is difficult to replicate the problem. I realize this is not helpful for fixing the Python memory manager, and I wish I could be more helpful. However, replacing PyObject_New with malloc does solve the problem for us and that may help anybody else in this situation in the future. Best regards, -Travis _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com