rockhowse commented on issue #11665: URL: https://github.com/apache/arrow/issues/11665#issuecomment-1006965295
Did some extensive testing of this scenario on both Windows 10 (32GB and 128GB RAM machines) as well as MacOSX (32GB) machine. I tested all using `R 4.1.2` and `arrow 6.0.1` as those are the latest stable versions of R and the arrow module from CRAN. The most consistent producer of the scenario was the Windows 10 64GB machine using default Page File configuration. All testing steps, observations and screenshots of RStudio + Memory/SWP consumption while testing I popped into this github repo: https://github.com/rockhowse/apache_arrow_32GB_write_feather Prime example of the state you can get into even with 128GB of RAM and the provided R code is shown on [this screenshot](https://github.com/rockhowse/apache_arrow_32GB_write_feather/blob/main/windows_10_21H2_128GB_RAM_2TB_nVME/screenshots/004_01_feather_128GB_RAM_hang_0_CPU_0_disk.PNG). Here's what I see occurring: - DF is created successfully, gobbling up between 32-64GB of RAM - fwrite completes successfully with a ~44GB file written out peaking out at/around ~68-70GB of RAM used - feather write steadily increases in RAM in a period with no disk activity (memcpy?) even hitting 100% on the 128GB box - At this point it can go two ways: - completes successfully with ~14GB feather file - gets into a state after hitting ~100% RAM consumption where CPU and disk usage drop to 0% and the application "hangs" without error - If it get into a hung state, you have to force kill RStudio, [I have gotten dialogs](https://github.com/rockhowse/apache_arrow_32GB_write_feather/blob/main/windows_10_21H2_128GB_RAM_2TB_nVME/screenshots/011_03_term_RStudio_fails_to_respond.PNG) stating that the app isn't responding to the attempt to close it and have to do a force. Interestingly enough... when testing the same versions of `R` and `arrow` on MacOSX with a 32GB box it does in fact complete, but takes forever due to MacOSX having to constantly resize the SWP partition to accommodate the increase requirement for virtual memory. It peaks out at/around 32GB of SWP. I haven't yet re-configured my windows developer environment on my new rig, but will set it up and see what the debugger has to say. My gut feeling is that we are in a `DOM` vs `SAX` type scenario where the feather write implementation as it's currently being consumed in `R` is doing a lot more memory consumption than might be obvious with smaller data sets. Given there is a large period of time where there is no disk IO but lots of CPU and memory it would lead me to believe it's not "streaming" as effectively as it could be. Hopefully this is helpful. I used this issue as a way for me to get back up to speed with arrow as it's been a couple years. I think I will start by using R debugging as suggested by @thisisnic. I am not well versed in lower level debugging in R, but if I can hook into a debug build of the native arrow lib being used, it should open up a bit more visibility into what's going on under the hood during that call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
