Travis Oliphant <[EMAIL PROTECTED]> wrote: > > [EMAIL PROTECTED] wrote: > > > Travis> More to the point, however, these scalar objects were allocated > > Travis> using the standard PyObject_New and PyObject_Del functions which > > Travis> of course use the Python memory manager. One user ported his > > Travis> (long-running) code to the new scipy core and found much to his > > Travis> dismay that what used to consume around 100MB now completely > > Travis> dominated his machine consuming up to 2GB of memory after only a > > Travis> few iterations. After searching many hours for memory leaks in > > Travis> scipy core (not a bad exercise anyway as some were found), the > > Travis> real problem was tracked to the fact that his code ended up > > Travis> creating and destroying many of these new array scalars. > > > >What Python object were his array elements a subclass of? > > These were all scipy core arrays. The elements were therefore all > C-like numbers (floats and integers I think). If he obtained an element > in Python, he would get an instance of a new "array" scalar object which > is a builtin extension type written in C. The important issue though is > that these "array" scalars were allocated using PyObject_New and > deallocated using PyObject_Del. The problem is that the Python memory > manager did not free the memory.
This is not a bug, and there doesn't seem to be any plans to change the behavior: python.org/sf/1338264 If I remember correctly, arrays from the Python standard library (import array), as well as numarray and Numeric, all store values in their pure C representations (they don't use PyObject_New unless someone uses the Python interface to fetch a particular element). This saves the overhead of allocating base objects, as well as the 3-5x space blowup when using Python integers (depending on whether your platform has 32 or 64 bit ints). > I think definitely, his usage pattern represented a "bad" corner case. > An unusable "corner" case in fact. At any rate, moving to use the > system free and malloc fixed the immediate problem. I mainly wanted to > report the problem here just as another piece of anecdotal evidence. On the one hand, using PyObjects embedded in an array in scientific Python is a good idea; you can use all of the standard Python manipulations on them. On the other hand, other similar projects have found it more efficient to never embed PyObjects in their arrays, and just allocate them as necessary on access. - Josiah _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
