[EMAIL PROTECTED] wrote: > Travis> More to the point, however, these scalar objects were allocated > Travis> using the standard PyObject_New and PyObject_Del functions which > Travis> of course use the Python memory manager. One user ported his > Travis> (long-running) code to the new scipy core and found much to his > Travis> dismay that what used to consume around 100MB now completely > Travis> dominated his machine consuming up to 2GB of memory after only a > Travis> few iterations. After searching many hours for memory leaks in > Travis> scipy core (not a bad exercise anyway as some were found), the > Travis> real problem was tracked to the fact that his code ended up > Travis> creating and destroying many of these new array scalars. > >What Python object were his array elements a subclass of? > > These were all scipy core arrays. The elements were therefore all C-like numbers (floats and integers I think). If he obtained an element in Python, he would get an instance of a new "array" scalar object which is a builtin extension type written in C. The important issue though is that these "array" scalars were allocated using PyObject_New and deallocated using PyObject_Del. The problem is that the Python memory manager did not free the memory.
> Travis> In the long term, what is the status of plans to re-work the > Travis> Python Memory manager to free memory that it acquires (or > Travis> improve the detection of already freed memory locations). > >None that I'm aware of. It's seen a great deal of work in the past and >generally doesn't cause problems. Maybe your user's usage patterns were >a bad corner case. It's hard to tell without more details. > > I think definitely, his usage pattern represented a "bad" corner case. An unusable "corner" case in fact. At any rate, moving to use the system free and malloc fixed the immediate problem. I mainly wanted to report the problem here just as another piece of anecdotal evidence. -Travis _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
