d33bs opened a new issue, #37361: URL: https://github.com/apache/arrow/issues/37361
### Describe the usage question you have. Please include as many useful details as possible. Thank you for the fantastic work on the Arrow project! I wanted to open this issue to address how memory allocators are selected for use more generally and also for PyArrow specifically (if this somehow differs). My understanding is that PyArrow may select to use `malloc` (labeled `system` in the context of Arrow memory allocator selection), `jemalloc`, or `mimalloc` depending on the operating system and allocator availability. In my experience on MacOS specifically I found that `mimalloc` is oftentimes selected by default. Investigating for reasoning on this default led me to [this article on R benchmarks from Ursa Labs](https://ursalabs.org/blog/2021-r-benchmarks-part-1/) and these notes from a [Arrow v4.0.0.x release in NEWS.md](https://github.com/apache/arrow/blob/b9453a2a33e935e26dc8358846ad671aefdcb4d0/r/NEWS.md?plain=1#L670). Based on this, the work towards benchmarking and `mimalloc` selection here seemed to pertain to early decisions in the Arrow project surrounding Arrow CSV read performance and possible bugs in `jemalloc` at the time of v4.0.0.x release. Recently I've been investigating PyArrow performance in an effort to better understand how much memory is consumed when reading parquet files. In profiling this I noticed that `mimalloc` consistently consumes more memory than `malloc` or `jemalloc`. Some of this may be seen from [a matrixed test through GitHub Actions](https://github.com/d33bs/CytoTable-benchmarks/actions/runs/5965146631). For example, Python 3.10 with the latest MacOS image showed memory totals of `mimalloc`: 722.037MB ([ref](https://github.com/d33bs/CytoTable-benchmarks/actions/runs/5965146631/job/16181914007)) and `jemalloc`: 657.041MB ([ref](https://github.com/d33bs/CytoTable-benchmarks/actions/runs/5965146631/job/16181912941)). This made me wonder about default memory allocator selection per operating system and how much risk there may be in overriding the default. Candidly, I recognize the "small" nature of my own testing so this question partially comes from a standpoint of better understanding how and where Arrow memory allocator selection takes place so as to improve implementation. Specifically I'm wondering: - What aspects influence default memory allocator selection in PyArrow (is this dynamic)? - When overriding default memory allocator selection, what risks should developers be aware of? To reproduce these results, please see the following: **Data generation:** ```python """ Generate two parquet files with identical random number data. """ import shutil import numpy as np import pyarrow as pa from pyarrow import parquet # create randomized number data and related pyarrow table tbl_numeric = pa.Table.from_arrays( [pa.array(np.random.rand(1000, 100)[:, i]) for i in range(100)], names=[f"Column_{i}" for i in range(100)], ) # Create a table and write it to file parquet.write_table( table=tbl_numeric, where="./random_number_data.parquet", ) # create a duplicate file for use in looped testing shutil.copy( "./random_number_data.parquet", "./random_number_data-copy.parquet", ) ``` **PyArrow parquet reads to profile (as `pyarrow_parquet_reads.py`)** ```python """ An example to test pyarrow reads using parquet files. """ import pyarrow as pa from pyarrow import parquet for parquet_file in [ "./random_number_data.parquet", "./random_number_data-copy.parquet", ]: parquet.read_table(source=parquet_file) ``` **[Memray](https://bloomberg.github.io/memray/index.html) profiling commands** Note: I used the environment variable ARROW_DEFAULT_MEMORY_POOL set to `system`, `jemalloc`, or `mimalloc` as per [Arrow documentation](https://arrow.apache.org/docs/cpp/env_vars.html#envvar-ARROW_DEFAULT_MEMORY_POOL). ```shell # run memray profiling memray run --output pyarrow_parquet_reads.py.memray.bin --force ./pyarrow_parquet_reads.py # display stats from memray profiling memray stats pyarrow_parquet_reads.py.memray.bin ``` ### Component(s) Benchmarking, Parquet, Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
