More detailed information:
## With annotations
=== tracemalloc stat ===
traced: (46969277, 46983753)
18,048,888 / 181112
File "<frozen importlib._bootstrap_external>", line 488
File "<frozen importlib._bootstrap_external>", line 780
File "<frozen importlib._bootstrap_external>", line 675
=== size by types ===
dict 9,083,816 (8,870.91KB) / 21846 = 415.811bytes (21.38%)
tuple 6,420,960 (6,270.47KB) / 86781 = 73.990bytes (15.11%)
str 6,050,213 (5,908.41KB) / 77527 = 78.040bytes (14.24%)
function 2,772,224 (2,707.25KB) / 20384 = 136.000bytes (6.53%)
code 2,744,888 (2,680.55KB) / 18987 = 144.567bytes (6.46%)
type 2,713,552 (2,649.95KB) / 2769 = 979.975bytes (6.39%)
bytes 2,650,838 (2,588.71KB) / 38723 = 68.456bytes (6.24%)
set 2,445,280 (2,387.97KB) / 6969 = 350.880bytes (5.76%)
weakref 1,255,600 (1,226.17KB) / 15695 = 80.000bytes (2.96%)
list 707,336 (690.76KB) / 6628 = 106.719bytes (1.66%)
=== dict stat ===
t, size, total (%) / count
3, 256, 1,479,424 (15.68%) / 5779.0
3, 1,200, 1,330,800 (14.11%) / 1109.0
3, 1,310,832, 1,310,832 (13.90%) / 1.0
3, 664, 1,287,496 (13.65%) / 1939.0
7, 128, 756,352 (8.02%) / 5909.0
3, 384, 707,328 (7.50%) / 1842.0
3, 2,296, 642,880 (6.81%) / 280.0
0, 256, 378,112 (4.01%) / 1477.0
7, 168, 251,832 (2.67%) / 1499.0
3, 4,720, 221,840 (2.35%) / 47.0
3, 9,336, 130,704 (1.39%) / 14.0
7, 88, 105,072 (1.11%) / 1194.0
* t=7 key-sharing dict, t=3 interned string key only, t=1 string key
only, t=0 non string key is used
## Stripped annotations
=== tracemalloc stat ===
traced: (42383739, 42397983)
18,069,806 / 181346
File "<frozen importlib._bootstrap_external>", line 488
File "<frozen importlib._bootstrap_external>", line 780
File "<frozen importlib._bootstrap_external>", line 675
=== size by types ===
dict 7,913,144 (7,727.68KB) / 17598 = 449.662bytes (20.62%)
tuple 6,149,120 (6,005.00KB) / 82734 = 74.324bytes (16.02%)
str 6,070,083 (5,927.82KB) / 77741 = 78.081bytes (15.82%)
code 2,744,312 (2,679.99KB) / 18983 = 144.567bytes (7.15%)
type 2,713,552 (2,649.95KB) / 2769 = 979.975bytes (7.07%)
bytes 2,650,464 (2,588.34KB) / 38715 = 68.461bytes (6.91%)
function 2,547,280 (2,487.58KB) / 18730 = 136.000bytes (6.64%)
set 1,423,520 (1,390.16KB) / 4627 = 307.655bytes (3.71%)
list 634,472 (619.60KB) / 5454 = 116.331bytes (1.65%)
int 608,784 (594.52KB) / 21021 = 28.961bytes (1.59%)
=== dict stat ===
t, size, total (%) / count
3, 1,200, 1,316,400 (16.06%) / 1097.0
3, 1,310,832, 1,310,832 (16.00%) / 1.0
3, 664, 942,216 (11.50%) / 1419.0
3, 256, 861,184 (10.51%) / 3364.0
3, 384, 657,024 (8.02%) / 1711.0
3, 2,296, 640,584 (7.82%) / 279.0
7, 128, 606,464 (7.40%) / 4738.0
0, 256, 379,904 (4.64%) / 1484.0
7, 168, 251,832 (3.07%) / 1499.0
3, 4,720, 221,840 (2.71%) / 47.0
3, 9,336, 130,704 (1.59%) / 14.0
7, 88, 105,248 (1.28%) / 1196.0
7, 256, 86,784 (1.06%) / 339.0
## Stripped annotation + without pydebug
=== tracemalloc stat ===
traced: (37371660, 40814265)
9,812,083 / 111082
File "<frozen importlib._bootstrap>", line 205
File "<frozen importlib._bootstrap_external>", line 742
File "<frozen importlib._bootstrap_external>", line 782
6,761,207 / 85614
File "<frozen importlib._bootstrap_external>", line 488
File "<frozen importlib._bootstrap_external>", line 780
File "<frozen importlib._bootstrap_external>", line 675
## Ideas about memory optimize
a) Split PyGC_Head from object
Reduces 2words (16byte) from each tuples.
>>> 82734 * 16 / 1024
1292.71875
So estimated -1.2MB
b) concat co_consts, co_names, co_varnames, co_freevars into one
tuple, or embed them into code.
Each tuple has 3 (gc head) + 3 (refcnt, *type, length) = 6 words
overhead. (or 4 words if (a) is applied)
If we can reduce 3 tuples, 18 words = 144byte (or 12 words=96byte) can
be reuduced.
>>> 18983 * 144
2733552
>>> 18983 * 96
1822368
But co_freevars is empty tuple in most cases. So real effect is
smaller than 2.7MB.
If we can embed them into code object, we can estimate -2.7MB.
(There are co_cellvars too. But I don't know about it much, especially
it is GC tracked or not)
c) (interned) string key only dict.
20% of memory is used for dict, and 70% of dict has interned string keys.
Current DictKeyEntry is 3 words: {key, hash, value}.
But if we can assume all key is string, hash can be get from the key.
If we use 2 words entry: {key, value} for such dict, I think dict can
be 25% smaller.
>>> 7913144 * 0.25 / 1024
1931.919921875
So I estimate -1.9MB
If we can implement (a)~(c) I estimate memory usage on Python
(--without-pydebug)
can be reduced from 35.6MB to 30MB, roughly.
But I think -Onoannotation option is top priority. It can reduce 4MB,
even we use
annotations only in our code.
If major libraries start using annotations, this number will be larger.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
