[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Serhiy Storchaka added the comment: Great work Inada-san! Thank you for your investigations! PR 13036 increases the chance of using borrowed references during pickling. Since this bug exists in the current code too (it just can be exposed in smaller number of cases), it should be fixed in

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Inada Naoki : -- nosy: +pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Inada Naoki added the comment: I'm using 1/10 version of dump.py I removed total_size() because it creates some garbages. sys._debugmallocstats() after load.py: master: # arenas allocated total =1,223 # arenas reclaimed =5 #

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Serhiy Storchaka added the comment: PR 13036 makes the C implementation no longer memoizing temporary objects. This decreases memory fragmentation and peak memory consumption on pickling and unpickling. -- ___ Python tracker

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Serhiy Storchaka : -- keywords: +patch pull_requests: +12956 stage: -> patch review ___ Python tracker ___ ___

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Inada Naoki added the comment: I confirmed this fragmentation is caused by memo in Unpickler. Pickler memos "reduce"-ed tuples while it is just a temporary object. I am not sure that this behavior is good. -- ___ Python tracker

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Inada Naoki added the comment: Memory allocation pattern is: alloc 24 # float alloc 24 alloc 24 alloc 64 # temporary tuple alloc 72 free 64 # free temporary tuples free 64 free 64 This cause some sort of fragmentation. Some pools in arenas are unused. This prevents pymalloc to return

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Inada Naoki : -- versions: +Python 3.8 -Python 3.6 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Inada Naoki : -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Paul Ellenbogen : Removed file: https://bugs.python.org/file48278/dump.py ___ Python tracker ___ ___ Python-bugs-list mailing

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Paul Ellenbogen : Removed file: https://bugs.python.org/file48281/dump.py ___ Python tracker ___ ___ Python-bugs-list mailing

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Paul Ellenbogen : Added file: https://bugs.python.org/file48282/dump.py ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Paul Ellenbogen added the comment: Good point. I have created a new version of dump that uses random() instead. float reuse explains the getsizeof difference, but there is still a significant memory usage difference. This makes sense to me because the original code I saw this issue in is

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Serhiy Storchaka added the comment: The difference is because in the first case all floats are the same float object 0.0, but in the second case they are different objects. For more reaĺistic comparison use different floats (for example random()). -- nosy: +serhiy.storchaka

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Paul Ellenbogen : Added file: https://bugs.python.org/file48280/common.py ___ Python tracker ___ ___ Python-bugs-list mailing

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

New submission from Paul Ellenbogen : Python encounters significant memory fragmentation when unpickling many small objects. I have attached two scripts that I believe demonstrate the issue. When you run "dumpy.py" it will generate a large list of namedtuples, then write that list to a file

[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

Change by Paul Ellenbogen : Added file: https://bugs.python.org/file48279/load.py ___ Python tracker ___ ___ Python-bugs-list mailing list