vminfant opened a new issue, #36100: URL: https://github.com/apache/arrow/issues/36100
### Describe the bug, including details regarding any error messages, version, and platform. Hi, I have been using pyarrow `Table.from_pylist` to convert a list to a pyarrow table and i have observed that the memory allocated by `parrow` doesn't get released until the program terminates (especially for **linux os**). My use case often involve long running processes and it is creating a problem. Sample script to reproduce the problem. I have used [memray](https://bloomberg.github.io/memray/getting_started.html) to profile and check the memory usage. ```python #file_name: test_exec.py import pyarrow as pa import time import random import string def get_sample_data(): record1 = {} for col_id in range(15): record1[f"column_{col_id}"] = string.ascii_letters[10 : random.randint(17, 49)] return [record1] def construct_data(data): count = 1 while count < 10: pa.Table.from_pylist(data * 100000) count += 1 return True def main(): data = get_sample_data() construct_data(data) print("construct data completed!") if __name__ == "__main__": main() time.sleep(180) ``` I have also checked the documentation and realised that there are different memory pool options can be used by setting an environment variable `ARROW_DEFAULT_MEMORY_POOL` to either `jemalloc`, `mimalloc` or `system` and i have tried all of them in both `macOS` and `linux` but I could see minor improvements on `macOS` but I couldn't see differences irrespective of the memory pool configurations on `linux`. `ARROW_DEFAULT_MEMORY_POOL=jemalloc` / OS: `macOS`  `ARROW_DEFAULT_MEMORY_POOL=jemalloc` / OS: `linux`  `ARROW_DEFAULT_MEMORY_POOL=mimalloc` / OS: `macOS`  `ARROW_DEFAULT_MEMORY_POOL=mimalloc` / OS: `linux`  `ARROW_DEFAULT_MEMORY_POOL=system` / OS: `macOS`  `ARROW_DEFAULT_MEMORY_POOL=system` / OS: `linux`  It will be really helpful if someone can highlight whats happening or suggest any approaches to eliminate this problem on linux. Highly appreciate your efforts on this issue. ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
