On Wed, Feb 27, 2019 at 9:59 PM Victor Stinner wrote:
> Maybe pickle is inefficient in its memory management and causes a lot
> of memory fragmentation?
>
No, it is not relating to pickle efficiency and memory fragmentation.
This problem is happened because pymalloc doesn't have no hysteresis
Maybe pickle is inefficient in its memory management and causes a lot
of memory fragmentation?
It's hard to write an efficient memory allocator :-( My notes on memory:
* "Excessive peak memory consumption by the Python parser"
https://bugs.python.org/issue26415
*
It happened very accidentally. Since venv is used,
many paths in the interpreter is changed. So how memory
is used are changed.
Let's reproduce the accident.
$ cat m2.py
import pickle, sys
LIST = pickle.dumps([[0]*10 for _ in range(10)], pickle.HIGHEST_PROTOCOL)
N = 1000
z = [[0]*10 for _ in
Sorry, I didn't get a coffee yet: more *often* in a venv.
Le mer. 27 févr. 2019 à 11:32, Victor Stinner a écrit :
>
> Any idea why Python calls mmap+munmap more even in a venv?
>
> Victor
>
> Le mer. 27 févr. 2019 à 10:00, INADA Naoki a écrit :
> >
> > >
> > > > Ah, another interesting point,
Any idea why Python calls mmap+munmap more even in a venv?
Victor
Le mer. 27 févr. 2019 à 10:00, INADA Naoki a écrit :
>
> >
> > > Ah, another interesting point, this huge slowdown happens only when
> > > bm_pickle.py
> > > is executed through pyperformance. When run it directly, slowdown is
>
> > Ah, another interesting point, this huge slowdown happens only when
bm_pickle.py
> > is executed through pyperformance. When run it directly, slowdown is
> > not so large.
>
> pyperformance runs benchmarks in a virtual environment. I don't know
> if it has any impact on bm_pickle.
>
> Most
