wjones127 commented on PR #33731: URL: https://github.com/apache/arrow/pull/33731#issuecomment-1416472513
I have good news and bad news. Bad news is that most high-level functionality (CSV, Parquet, Acero) has at least some memory pool allocations that just go directly to the default memory pool. So I'm not able to add memory benchmarks to them using this method yet. The only two benchmarks that we can measure today are the builders and the CSV converter. 😢 Good news is the by adding the `num_allocations` metric on memory pools, we can now write tests that assert all allocations happened on the passed memory pool. Basically just assert that `ProxyMemoryPool.num_allocations()` is equal to `default_memory_pool()->num_allocations()` minus the existing allocation count when the proxy pool was initialized. 🥳 I propose that we merge this, and in follow-up PRs we iteratively fix the memory pool handling in features and enable the memory benchmarks as we go. Of course, we should also add tests as described above to make sure we don't get regressions in the memory pool handling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
