[issue32759] multiprocessing.Array do not release shared memory

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 857d573257ab150d6bea666354312a5fd284b171 by Victor Stinner in 
branch '3.7':
bpo-39932: Fix multiprocessing test_heap() (GH-19690)
https://github.com/python/cpython/commit/857d573257ab150d6bea666354312a5fd284b171


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Yes, it's a minimal effort.  More sophisticated behavior would require a more 
sophisticated allocator.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This change looks safe to me. I was just not sure that it is enough for 
practical cases. What if after allocating a large buffer the rest of the new 
area would be used for allocating small buffers? They can keep references to 
the large area after freeing the large buffer. Perhaps it is worth to block 
marking the remainder of a large area available. This will increase the memory 
consumption by small percent, but will reduce the risk of prolonging the life 
time of large blocks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I pushed a fix for this in 3.8.  Since the fix is a bit delicate, I'd rather 
not backport it.  Thank you for reporting this issue!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset e4679cd644aa19f9d9df9beb1326625cf2b02c15 by Antoine Pitrou in 
branch 'master':
bpo-32759: Free unused arenas in multiprocessing.heap (GH-5827)
https://github.com/python/cpython/commit/e4679cd644aa19f9d9df9beb1326625cf2b02c15


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Antoine Pitrou]

Change by Antoine Pitrou :


--
keywords: +patch
pull_requests: +5605
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

It is also possible to uncommit the memory without deallocating it, making 
reuse potentially faster, but that requires low-level platform-specific code 
(e.g. madvise(MADV_DONTNEED) on Linux or DiscardVirtualMemory() on Windows).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Ok, this is because the multiprocessing Heap object never releases any unused 
arena objects, so the shared memory you allocate will probably stay allocated 
until the process tree ends.

It is possible to change the strategy to delete unused arenas, though it's 
unsure whether doing so has adverse consequences (such as making later 
allocations costlier).  It may make sense to only reclaim the larger arenas 
(larger than 1MB perhaps?).

--
versions: +Python 3.8 -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[Steve Dower]

Change by Steve Dower :


--
nosy: +davin, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[OO O]

OO O  added the comment:

mp.heap.BufferWrapper._heap = mp.heap.Heap ()
gc.collect ()

This is working!! The memory is cleared.
Just delete the globe _heap and recreate a new one, but is the the correct 
way???

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[OO O]

Change by OO O :


--
versions: +Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32759] multiprocessing.Array do not release shared memory

[OO O]

New submission from OO O :

OS: Win10 / 8.1
Python: 3.5 / 3.6

My program use mp.Array to share huge data.
But suffer from out of memory after running for a while. 

But Windows task manager didn't show which process use that huge memory. And I 
use pympler to check my python memory usage. Still shows noting. So, I use 
RamMap to check, it shows a huge shared memory is used.

I can reproduce the case by the simple test code:
   #---
   import numpy as np
   import multiprocessing as mp
   import gc

   def F ():
   a = mp.Array ( 'I', 18, lock = False )

   #
   F ()
   gc.collect ()
   #---

No matter how hard I tried. the memory is not released.
I put what I tried in the attachment picture.

--
components: Windows
files: result.jpg
messages: 311571
nosy: OO O, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: multiprocessing.Array do not release shared memory
type: resource usage
versions: Python 3.6
Added file: https://bugs.python.org/file47424/result.jpg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com