On 06/02/2017 02:46 AM, Victor Stinner wrote:
I would be curious of another test: use pymalloc for objects larger than 512 bytes. For example, allocate up to 4 KB? In the past, we already changed the maximum size from 256 to 512 to support most common Python objects on 64-bit platforms. Since Python objects contain many pointers: switching from 32 bit to 64 bit can double the size of the object in the worst case.
You've already seen Tim Peters' post about why we must leave pool size set to 4k. Obviously This in turn means using obmalloc for larger objects will mean more and more wasted memory.
For example, let's say we use obmalloc for allocations of 2048 bytes. Pool size is 4096 bytes, and there's a 48-byte "pool_header" structure on the front (on 64-bit platforms, if I counted right). So there are only 4048 bytes usable per pool. After the first 2048 allocation, we're left with 2000 bytes at the end. You can't use that memory for another allocation class, that's impossible given obmalloc's design. So that 2000 bytes is just wasted.
Currently obmalloc's maximum allocation size is 512 bytes; after 7 allocations, this leaves 464 wasted bytes at the end. Which isn't *great* exactly but it's only 11% of the overall allocated memory.
Anyway, I'm not super excited by the prospect of using obmalloc for larger objects. There's an inverse relation between the size of allocation and the frequency of allocation. In Python there are lots of tiny allocations, but fewer and fewer as the size increases. (A similarly-shaped graph to what retailers call the "long tail".) By no small coincidence, obmalloc is great at small objects, which is where we needed the help most. Let's leave it at that.
A more fruitful endeavor might be to try one of these fancy new third-party allocators in CPython, e.g. tcmalloc, jemalloc. Try each with both obmalloc turned on and turned off, and see what happens to performance and memory usage. (I'd try it myself, but I'm already so far behind on watching funny cat videos.)
//arry/
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
