[issue21220] Enhance obmalloc allocation strategy

Kristján Valur Jónsson added the comment: >> This significantly helps fragmentation in programs with dynamic memory >> usage, e.g. long running programs. > On which programs? The fragmentation of the memory depends a lot on how the > program allocates memory. For example, if a program has no "

[issue21220] Enhance obmalloc allocation strategy

STINNER Victor added the comment: I spend some nights to try to understand the memory usage of the following Python script: https://bitbucket.org/haypo/misc/src/31bf03ace91db3998981ee56caf80f09c29991f5/memory/python_memleak.py?at=default It looks like the weird memory usage (aka "memory fragmen

[issue21220] Enhance obmalloc allocation strategy

Kristján Valur Jónsson added the comment: Sure. I'm flying home from PyCon this afternoon. I´ll produce and tabulate data once I'm home on my workstation again. -- ___ Python tracker

[issue21220] Enhance obmalloc allocation strategy

Antoine Pitrou added the comment: > sorry, I meant of course "performance of pybench.py goes up" pybench is really a very poor (extremely low-level) benchmark. It would be nice if you could test with the benchmark suite at http://hg.python.org/benchmarks --

[issue21220] Enhance obmalloc allocation strategy

Kristján Valur Jónsson added the comment: sorry, I meant of course "performance of pybench.py goes up" -- ___ Python tracker ___ ___ P

[issue21220] Enhance obmalloc allocation strategy

Kristján Valur Jónsson added the comment: Antoine: The location of the arenas when they're individually allocated with mmap does not matter, no, but preferring to keep low address ones reduces vmem fragmentation, since they end up being clustered together in memory. For the usable-arenas lis

[issue21220] Enhance obmalloc allocation strategy

Charles-François Natali added the comment: > In Python 3, arenas are allocated using mmap(), so wherever the arena ends up > in the address space shouldn't matter, should it? Indeed, although the effect on cache locality isn't clear. Also, I don't think this solves the problem of having a singl

[issue21220] Enhance obmalloc allocation strategy

STINNER Victor added the comment: It would also be interesting to compare fragmentation and performances of Python with and without pymalloc, maybe with other heap allocators like FreeBSD jemalloc and Google TCMalloc. -- ___ Python tracker

[issue21220] Enhance obmalloc allocation strategy

STINNER Victor added the comment: It was also discussed to replace pymalloc with Windows Low Fragementation Heap (LFH) allocator on Windows: http://bugs.python.org/issue13483#msg148605 -- nosy: +haypo ___ Python tracker

[issue21220] Enhance obmalloc allocation strategy

Antoine Pitrou added the comment: In Python 3, arenas are allocated using mmap(), so wherever the arena ends up in the address space shouldn't matter, should it? Also, the new strategy adds an O(n) component in the allocator. -- nosy: +neologix, pitrou, tim.peters _

[issue21220] Enhance obmalloc allocation strategy

Kristján Valur Jónsson added the comment: Update patch with suggestions from Larry -- Added file: http://bugs.python.org/file34876/obmalloc.patch ___ Python tracker ___ _

[issue21220] Enhance obmalloc allocation strategy

Changes by Kristján Valur Jónsson : -- nosy: +larry ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue21220] Enhance obmalloc allocation strategy

New submission from Kristján Valur Jónsson: A new allocation policy, "lowest address strategy" improves fragmentation of memory in obmalloc. pools with available memory are chosen by lowest address preference. This increases the likelihood that unused pools are released to their correspondin