[issue32759] multiprocessing.Array do not release shared memory

STINNER Victor added the comment: New changeset 857d573257ab150d6bea666354312a5fd284b171 by Victor Stinner in branch '3.7': bpo-39932: Fix multiprocessing test_heap() (GH-19690) https://github.com/python/cpython/commit/857d573257ab150d6bea666354312a5fd284b171 -- nosy: +vstinner

[issue32759] multiprocessing.Array do not release shared memory

Antoine Pitrou added the comment: Yes, it's a minimal effort. More sophisticated behavior would require a more sophisticated allocator. -- ___ Python tracker

[issue32759] multiprocessing.Array do not release shared memory

Serhiy Storchaka added the comment: This change looks safe to me. I was just not sure that it is enough for practical cases. What if after allocating a large buffer the rest of the new area would be used for allocating small buffers? They can keep references to

[issue32759] multiprocessing.Array do not release shared memory

Antoine Pitrou added the comment: I pushed a fix for this in 3.8. Since the fix is a bit delicate, I'd rather not backport it. Thank you for reporting this issue! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed

[issue32759] multiprocessing.Array do not release shared memory

Antoine Pitrou added the comment: New changeset e4679cd644aa19f9d9df9beb1326625cf2b02c15 by Antoine Pitrou in branch 'master': bpo-32759: Free unused arenas in multiprocessing.heap (GH-5827) https://github.com/python/cpython/commit/e4679cd644aa19f9d9df9beb1326625cf2b02c15

[issue32759] multiprocessing.Array do not release shared memory

Change by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___

[issue32759] multiprocessing.Array do not release shared memory

Change by Antoine Pitrou : -- keywords: +patch pull_requests: +5605 stage: -> patch review ___ Python tracker ___

[issue32759] multiprocessing.Array do not release shared memory

Antoine Pitrou added the comment: It is also possible to uncommit the memory without deallocating it, making reuse potentially faster, but that requires low-level platform-specific code (e.g. madvise(MADV_DONTNEED) on Linux or DiscardVirtualMemory() on Windows). --

[issue32759] multiprocessing.Array do not release shared memory

Antoine Pitrou added the comment: Ok, this is because the multiprocessing Heap object never releases any unused arena objects, so the shared memory you allocate will probably stay allocated until the process tree ends. It is possible to change the strategy to delete unused

[issue32759] multiprocessing.Array do not release shared memory

Change by Steve Dower : -- nosy: +davin, pitrou ___ Python tracker ___ ___

[issue32759] multiprocessing.Array do not release shared memory

OO O added the comment: mp.heap.BufferWrapper._heap = mp.heap.Heap () gc.collect () This is working!! The memory is cleared. Just delete the globe _heap and recreate a new one, but is the the correct way??? -- ___ Python

[issue32759] multiprocessing.Array do not release shared memory

Change by OO O : -- versions: +Python 3.5 ___ Python tracker ___ ___ Python-bugs-list

[issue32759] multiprocessing.Array do not release shared memory

New submission from OO O : OS: Win10 / 8.1 Python: 3.5 / 3.6 My program use mp.Array to share huge data. But suffer from out of memory after running for a while. But Windows task manager didn't show which process use that huge memory. And I use pympler to check my python