asfimport opened a new issue, #31824: URL: https://github.com/apache/arrow/issues/31824
When looking at ARROW-15081 there was a strange amount of memory used even when we were accumulating all of the results into a single 64 byte counter (e.g. `SELECT COUNT(*) FROM table`). It turns out this was the parquet metadata, which gets attached to the parquet file fragment. There is no way to prevent this and, in this case, it was using quite a bit of RAM. There were 1100 files and each file had ~10MB of metadata. We should have an option for disabling this. Also, this should probably be off by default. It can be a useful thing to cache if you are going to run the same dataset again and again but otherwise it is just wasted RAM. **Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-16451) / @westonpace <sub>**Note**: *This issue was originally created as [ARROW-16451](https://issues.apache.org/jira/browse/ARROW-16451). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*</sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
