On 15 January 2016 at 05:25, Matthew Paulson <paul...@busiq.com> wrote:
> Hi All:
>
> I've created a simple program to make sure I wasn't lying to you all ;->
>
> Here it is:
>
> for (ii = 0; ii < 100; ii++)
> {
> Py_Initialize();
>
> if ((code = Py_CompileString(p, "foo", Py_file_input)) == NULL)
> printf("PyRun_SimpleString() failed\n");
> else
> {
> if (PyRun_SimpleString(p) == -1)
> printf("PyRun_SimpleString() failed\n");
>
> Py_CLEAR(code);
> }
>
> Py_Finalize();
> }
>
> This sequence causes about 10k growth per iteration and after many cycles,
> there's no indication that any pooling logic is helping. Our "useful"
> example is slightly more complex, and therefore may explain why I was
> seeing about 16k per iteration.
>
> Unless I've done something obviously wrong, I tend to believe Benjamin's
> claim that this issue is well known.
>
> Suggestion: I have had great success with similar problems in the past by
> using a pools implementation sitting on top of what I call a "block memory
> allocator". The bottom (block) allocator grabs large blocks from the heap
> and then doles them out to the pools layer, which in turn doles them out to
> the requester. When client memory is freed -- it is NOT -- rather it's
> added to the pool which contains like-sized blocks -- call it an "organized
> free list". This is a very, very fast way to handle high allocation
> frequency patterns. Finally, during shutdown, the pool simply vaporizes
> and the block allocator returns a the (fewer) large blocks back to the
> heap. This avoids thrashing the heap, forcing it to coalesce inefficiently
> and also avoids heap fragmentation, which can cause unwanted growth as
> well...
>
> Note that this would be a "hard-reset" of all allocated memory, and any
> global data in the text segment would also have to be cleared, but it would
> provide a fast, clean way to ensure that each invocation was 100% clean.
>
CPython does use an arena based allocator, but PyFinalize doesn't purge it
(if it did, there'd be segfaults rather than memory growth when modules
keep pointers across Initialize/Finalize cycles).
Building with PYMALLOC_DEBUG and setting PYTHONMALLOCSTATS in the
environment will cause it to dump debugging info during Py_Finalize.
Building with Py_TRACE_REFS and setting PYTHONDUMPREFS also provides info
on live Python objects during shutdown.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
