On 2019-06-09, Tim Peters wrote:
> And now there's a PR that removes obmalloc's limit on pool sizes, and,
> for a start, quadruples pool (and arena!) sizes on 64-bit boxes:
Neat.
> As the PR says,
>
> """
> It would be great to get feedback from 64-bit apps that do massive
> amounts of small-object allocations and deallocations.
> """
I've done a little testing the pool overhead. I have an application
that uses many small dicts as holders of data. The function:
sys._debugmallocstats()
is useful to get stats for the obmalloc pools. Total data allocated
by obmalloc is 262 MB. At the 4*PAGE_SIZE pool size, the wasted
space due to partly filled pools is only 0.18%. For 16*PAGE_SIZE
pools, 0.71%.
I have a set of stats for another program. In that case, total
memory allocated by obmalloc is 14 MB. For 4*PAGE_SIZE pools,
wasted space is 0.78% of total. At 16*PAGE_SIZE, it is 2.4%.
Based on that small set of data, using 4*PAGE_SIZE seems
conservative. As I'm sure you realize, making pools bigger will
waste actual memory, not just virtual address space because you
write the arena pointer to each OS page.
I want to do performance profiling using Linux perf. That should
show where the hotspot instructions in the obmalloc code. Maybe
that will be useful to you.
Another thought about address_in_range(): some operating systems
allow you to allocate memory a specific alignments. Or, you can
even allocate a chunk of memory at a fixed memory location if you do
the correct magic incantation. I noticed that Go does that. I
imagine doing that has a bunch of associated challenges with it.
However, if we could control the alignment and memory location of
obmalloc arenas, we would not have the segv problem of
address_in_range(). It's probably not worth going down that path
due to the problems involved.
Regards,
Neil
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/LJWXNQNZRVCCF36ALXKZVWYHINXTVMVB/
