I know (thanks to Google) that much has been said in the past about the Python Memory Manager. My purpose in posting is simply to given a use-case example of how the current memory manager (in Python 2.4.X) can be problematic in scientific/engineering code.
Scipy core is a replacement for Numeric. One of the things scipy core does is define a new python scalar object for ever data type that an array can have (currently 21). This has many advantages and is made feasible by the ability of Python to subtype in C. These scalars all inherit from the standard Python types where there is a correspondence. More to the point, however, these scalar objects were allocated using the standard PyObject_New and PyObject_Del functions which of course use the Python memory manager. One user ported his (long-running) code to the new scipy core and found much to his dismay that what used to consume around 100MB now completely dominated his machine consuming up to 2GB of memory after only a few iterations. After searching many hours for memory leaks in scipy core (not a bad exercise anyway as some were found), the real problem was tracked to the fact that his code ended up creating and destroying many of these new array scalars. The Python memory manager was not reusing memory (even though PyObject_Del was being called). I don't know enough about the memory manager to understand why that was happening. However, changing the allocation from PyObject_New to malloc and from PyObject_Del to free, fixed the problems this user was seeing. Now the code runs for a long time consuming only around 100MB at-a-time. Thus, all of the objects in scipy core now use system malloc and system free for their memory needs. Perhaps this is unfortunate, but it was the only solution I could see in the short term. In the long term, what is the status of plans to re-work the Python Memory manager to free memory that it acquires (or improve the detection of already freed memory locations). I see from other postings that this has been a problem for other people as well. Also, is there a recommended way for dealing with this problem other than using system malloc and system free (or I suppose writing your own specialized memory manager). Thanks for any feedback, -Travis Oliphant _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
