I guess that INADA-san used pyperformance --track-memory. pyperf --track-memory doc: "--track-memory: get the memory peak usage. it is less accurate than tracemalloc, but has a lower overhead. On Linux, compute the sum of Private_Clean and Private_Dirty memory mappings of /proc/self/smaps. On Windows, get PeakPagefileUsage of GetProcessMemoryInfo() (of the current process): the peak value of the Commit Charge during the lifetime of this process." https://pyperf.readthedocs.io/en/latest/runner.html#misc
On Linux, pyperf uses read_smap_file() of pyperf._memory: https://github.com/vstinner/pyperf/blob/master/pyperf/_memory.py # Code to parse Linux /proc/%d/smaps files. # # See http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html for # a quick introduction to smaps. # # Need Linux 2.6.16 or newer. def read_smap_file(): total = 0 fp = open(proc_path("self/smaps"), "rb") with fp: for line in fp: # Include both Private_Clean and Private_Dirty sections. line = line.rstrip() if line.startswith(b"Private_") and line.endswith(b'kB'): parts = line.split() total += int(parts[1]) * 1024 return total It spawns a thread which reads /proc/self/smaps every milliseconds and then report the *maximum*. Victor Le jeu. 4 juil. 2019 à 17:12, Tim Peters <tim.pet...@gmail.com> a écrit : > > [Antoine Pitrou <solip...@pitrou.net>] > >> Ah, interesting. Were you able to measure the memory footprint as well? > > [Inada Naoki <songofaca...@gmail.com>] > > Hmm, it is not good. mimalloc uses MADV_FREE so it may affect to some > > benchmarks. I will look it later. > > > > ``` > > $ ./python -m pyperf compare_to pymalloc-mem.json mimalloc-mem.json -G > > Slower (60): > > - logging_format: 10.6 MB +- 384.2 kB -> 27.2 MB +- 21.3 kB: 2.58x > > slower (+158%) > > ... > > Could you say more about what's being measured here? Like, if this is > on Linux, is it reporting RSS? VSS? Something else? > > mimalloc is "most like" obmalloc among those I've looked at in recent > weeks. I noted before that its pools (they call them "pages") and > arenas (called "segments") are at least 16x larger than obmalloc uses > (64 KiB minimum pool/page size, and 4 MiB minimum arena/segment size, > in mimalloc). > > Like all "modern" 64-bit allocators, it cares little about reserving > largish blobs of address space, so I'd _expect_ Linuxy VSS to zoom. > But it also releases (in some sense, ;like MADV_FREE) physical RAM > back to the system at a granularity far smaller than arena'segment. > > At an extreme, the SuperMalloc I linked to earlier reserves a 512 MiB > vector at the start, so no program linking to that can consume less > than half a gig of address space. But it only expects to _use_ a few > 4 KiB OS pages out of that. mimalloc doesn't do anything anywhere > near _that_ gonzo (& since mimalloc comes out of Microsoft, it never > will - "total virtual memory" on Windows is a system-wide resource, > limited to the sum of physical RAM + pagefile size - no "overcommit" > is allowed). > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/ES3BFCBXS7N56XGUHHSOPHRT3UAEGKVA/ -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HF4UIZP5J3KKWQMLCHKJD3G6YZLHWWBE/
