[issue28509] dict.update allocates too much

Change by Serhiy Storchaka : -- pull_requests: -839 ___ Python tracker ___ ___

[issue28509] dict.update allocates too much

Changes by Donald Stufft : -- pull_requests: +839 ___ Python tracker ___ ___

[issue28509] dict.update allocates too much

Changes by INADA Naoki : -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker ___

[issue28509] dict.update allocates too much

Roundup Robot added the comment: New changeset 8c2615decd2e by INADA Naoki in branch '3.6': Issue #28509: dict.update() no longer allocate unnecessary large memory https://hg.python.org/cpython/rev/8c2615decd2e New changeset deb3e5857d8c by INADA Naoki in branch 'default': Issue #28509:

[issue28509] dict.update allocates too much

Serhiy Storchaka added the comment: 28509-smaller-update2.patch LGTM. Your idea in msg279480 looks worth, but needs benchmarking different corner cases to check that it doesn't cause to a regression. I think we will have enough time for this at 3.7 developing stage. -- assignee: ->

[issue28509] dict.update allocates too much

INADA Naoki added the comment: OK, I won't change it to allow additional resize while merging, after pre-resize. But current code has two problem: * Pre-resize happen even when resizing is not necessary. (ex. two dict has same keys). * Pre-resize allocates too much memory which doesn't make

[issue28509] dict.update allocates too much

Raymond Hettinger added the comment: FWIW, I prefer the existing code be left as is. We've already greatly compacted the dictionaries. There is no need to be ultra-aggressive in shaving off every little corner. There is some advantage to having the dict be more sparse (fewer collisions,

[issue28509] dict.update allocates too much

INADA Naoki added the comment: I feel that accept one resize while merging is better. How about this? /* Do one big resize at the start, rather than incrementally * resizing. At most one resize happen while merging. */ if (USABLE_FRACTION(mp->ma_keys->dk_size)

[issue28509] dict.update allocates too much

Serhiy Storchaka added the comment: LGTM. Here is a script that produces more compact output. The first column is a size after which the dict is resized. import sys p = 1 b = {} for i in range(1): a = {} b[i] = i a.update(b) s = sys.getsizeof(a) if s > p:

[issue28509] dict.update allocates too much

Changes by Serhiy Storchaka : -- stage: needs patch -> patch review ___ Python tracker ___

[issue28509] dict.update allocates too much

INADA Naoki added the comment: script: import sys for i in range(25): a = {} b = {j: j for j in range(i)} a.update(b) print(i, sys.getsizeof(a)) before: 0 256 1 256 2 256 3 256 4 256 5 256 6 384 7 384 8 664 9 664 10 664 11 664 12 664 13 664 14 664 15 664 16 1200 17 1200 18 1200

[issue28509] dict.update allocates too much

Serhiy Storchaka added the comment: > Dict is resized. And since __dict__.update() caused the resizing, both are normal (combined) dict. Ah, my bad, I missed this point. The original issue now disappears. Thanks Naoki. But dict.update (and maybe inserting) can use more economical allocation