vminfant opened a new issue, #36100:
URL: https://github.com/apache/arrow/issues/36100

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   Hi,
   
   I have been using pyarrow `Table.from_pylist` to convert a list to a pyarrow 
table and i have observed that the memory allocated by `parrow` doesn't get 
released until the program terminates (especially for **linux os**). My use 
case often involve long running processes and it is creating a problem. 
   
   Sample script to reproduce the problem. I have used 
[memray](https://bloomberg.github.io/memray/getting_started.html) to profile 
and check the memory usage. 
   
   ```python
   #file_name: test_exec.py
   
   import pyarrow as pa
   import time
   import random
   import string
   
   def get_sample_data():
       record1 = {}
       for col_id in range(15):
           record1[f"column_{col_id}"] = string.ascii_letters[10 : 
random.randint(17, 49)]
   
       return [record1]
   
   def construct_data(data):
       count = 1
       while count < 10:
           pa.Table.from_pylist(data * 100000)
           count += 1
       return True
   
   def main():
       data = get_sample_data()
       construct_data(data)
       print("construct data completed!")
   
   if __name__ == "__main__":
       main()
       time.sleep(180)
   ```
   
   I have also checked the documentation and realised that there are different 
memory pool options can be used by setting an environment variable 
`ARROW_DEFAULT_MEMORY_POOL` to either `jemalloc`, `mimalloc` or `system` and i 
have tried all of them in both `macOS` and `linux` but I could see minor 
improvements on `macOS` but I couldn't see differences irrespective of the 
memory pool configurations on `linux`. 
   
   `ARROW_DEFAULT_MEMORY_POOL=jemalloc` / OS: `macOS`
   
   
![image](https://github.com/apache/arrow/assets/27796304/dc8b56f9-2ed5-4106-b0c6-f6401ed69798)
   
   `ARROW_DEFAULT_MEMORY_POOL=jemalloc` / OS: `linux`
   
   
![image](https://github.com/apache/arrow/assets/27796304/4e7857dc-18b9-42af-ae32-a5399043692b)
   
   `ARROW_DEFAULT_MEMORY_POOL=mimalloc` / OS: `macOS`
   
   
![image](https://github.com/apache/arrow/assets/27796304/52a55f78-3778-4a17-8004-40bce1477aab)
   
   `ARROW_DEFAULT_MEMORY_POOL=mimalloc` / OS: `linux`
   
   
![image](https://github.com/apache/arrow/assets/27796304/78a3cd32-7bf4-4b48-afae-d03c1d746982)
   
   `ARROW_DEFAULT_MEMORY_POOL=system` / OS: `macOS`
   
   
![image](https://github.com/apache/arrow/assets/27796304/d6319b7c-aa94-4bf6-96ce-11cab9c13487)
   
   `ARROW_DEFAULT_MEMORY_POOL=system` / OS: `linux`
   
   
![image](https://github.com/apache/arrow/assets/27796304/0469b33b-7665-4a36-a14e-e696ba28b0ef)
   
   It will be really helpful if someone can highlight whats happening or 
suggest any approaches to eliminate this problem on linux. Highly appreciate 
your efforts on this issue.
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to