kou commented on issue #38275: URL: https://github.com/apache/arrow/issues/38275#issuecomment-1771905588
We need to load all `mmap`-ed contents to memory with the stream format + `read_all`. In your case, the contents are larger than all RAM. So it has performance penalty (such as disk I/O). If we use the file format, we just need to load only needed `mmap`-ed contents to memory. In your case, you don't need all contents. So you can process your data without loading all contents. I hope that this explanation helps you. (You may want to use `iotop`/`iostat`/... to monitor I/O while you process your data.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
