pitrou commented on issue #40301:
URL: https://github.com/apache/arrow/issues/40301#issuecomment-2046949855
Also a couple more observations:
* on Linux (Ubuntu 22.04), if you switch to the system allocator
(`ARROW_DEFAULT_MEMORY_POOL=system`) the behavior disappears, which is not
better in itself, but may reassure users
* if you choose mimalloc, `MIMALLOC_SHOW_STATS=1` gives you summary stats
_from mimalloc's PoV_ at shutdown.
We can therefore confirm that the amount of memory committed (not just
reserved) is reasonably small:
```console
$ ARROW_DEFAULT_MEMORY_POOL=mimalloc MIMALLOC_SHOW_STATS=1 python -c "import
pyarrow as pa; table = pa.table({'A': 'a', 'B': 'b', 'C': 'c', 'D': 'd', 'E':
'e', 'F': 'f', 'G': 'g', 'H':'h', 'I': 'i', 'J': 'j', 'K':'k', 'L':'l',
'M':'m', 'N':'n', 'O':'o', 'P':'p', 'Q':'q', 'R':'r', 'S':'s', 'T':'t',
'U':'u', 'V':'v', 'W':'w', 'X':'x','Y':'y','Z':'z'}); table.to_pandas()"
heap stats: peak total freed current unit count
reserved: 1.5 GiB 1.5 GiB 128.5 KiB 1.5 GiB
not all freed!
committed: 112.1 MiB 115.1 MiB 115.1 MiB 64.2 KiB
not all freed!
reset: 0 0 0 0
ok
touched: 3.1 MiB 3.1 MiB 3.2 MiB -100.3 KiB
ok
segments: 25 25 25 0
ok
-abandoned: 0 0 0 0
ok
-cached: 0 0 0 0
ok
pages: 0 0 25 -25
ok
-abandoned: 0 0 0 0
ok
-extended: 0
-noretire: 0
mmaps: 0
commits: 48
threads: 24 24 48 -24
ok
searches: 0.0 avg
numa nodes: 1
elapsed: 0.255 s
process: user: 0.232 s, system: 0.038 s, faults: 2, rss: 90.6 MiB,
commit: 112.1 MiB
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
