[GitHub] [arrow] GianlucaFicarelli commented on issue #32670: pyarrow may use a lot of memory to load a dataframe from parquet

[via GitHub]

GianlucaFicarelli commented on issue #32670:
URL: https://github.com/apache/arrow/issues/32670#issuecomment-1539640987

   Hi @mapleFU , I re-executed the initial script with pyarrow 12.0.0 and 6.0.1.
   I used py-spy and memory-profiler to produce the attached graphs.
   
   ```
   pip freeze  # pyarrow==12.0.0
   
   memory-profiler==0.61.0
   numpy==1.24.3
   pandas==2.0.1
   psutil==5.9.5
   py-spy==0.3.14
   pyarrow==12.0.0
   python-dateutil==2.8.2
   pytz==2023.3
   six==1.16.0
   tzdata==2023.3
   ```
   
   ```
   pip freeze  # pyarrow==6.0.1
   
   memory-profiler==0.61.0
   numpy==1.24.3
   pandas==2.0.1
   psutil==5.9.5
   py-spy==0.3.14
   pyarrow==6.0.1
   python-dateutil==2.8.2
   pytz==2023.3
   six==1.16.0
   tzdata==2023.3
   ```
   
   ```
   python try_arrow.py  # pyarrow==12.0.0
   
     0 time:       0.0 rss:     112.0
     1 time:       3.8 rss:    1220.9
     2 time:       6.0 rss:    1241.1
     3 time:       6.4 rss:     742.2
     4 time:       9.7 rss:     740.9
     5 time:      11.8 rss:    2194.6
     6 time:      14.8 rss:    1784.6
     7 time:      15.0 rss:    1289.3
     8 time:      18.0 rss:     907.3
   ```
   ```
   python try_arrow.py  # pyarrow==6.0.1
   
     0 time:       0.0 rss:      78.7
     1 time:       3.8 rss:    1187.1
     2 time:       5.9 rss:    1949.4
     3 time:       6.3 rss:     732.2
     4 time:       9.6 rss:     710.2
     5 time:      11.8 rss:    2171.3
     6 time:      14.8 rss:    1757.7
     7 time:      15.0 rss:    1261.5
     8 time:      18.0 rss:     879.5
   ```
   
   ```
   py-spy record --native --subprocesses python test_pyarrow.py
   ```
   
![python_12](https://user-images.githubusercontent.com/26835404/237032439-325e2189-8232-4294-8642-3620f8eab433.svg)
   
![python_6](https://user-images.githubusercontent.com/26835404/237032480-17b2f737-695b-4d87-8a44-05ae9598ab14.svg)
   
   
   ```
   mprof run --multiprocess python test_pyarrow.py
   ```
   
   <img width="1372" alt="mprofile_12" 
src="https://user-images.githubusercontent.com/26835404/237032521-7a62610a-1d83-4297-a33b-df4fb55fe3df.png";>
   
   <img width="1372" alt="mprofile_6" 
src="https://user-images.githubusercontent.com/26835404/237032582-2bc2c51d-c11b-47ee-ab9a-93b03cf38505.png";>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org