RussellSpitzer opened a new issue, #13937: URL: https://github.com/apache/iceberg/issues/13937
### Apache Iceberg version 1.9.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Related to #13935 Currently the code based has a Heap Memory leak in the ArrowAllocation.rootAllocator(). Every [VectorizedReadBuilder](https://github.com/apache/iceberg/blob/2b66fc553acc0e7ddc9edcdb26095daf4e8a04c2/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java#L75-L77) creates a new child allocator which is used to allocate vectors for that particular reader. These allocators are *never released* even if when we close all of the vectors we allocate from it. If we did close these, we would have seen the issues in #13935 way earlier since every application would end with a string of "MemoryLeak Detected" messages. I think we should probably do 2 things. 1) The root allocator from ArrowAllocation should be close during teardown 2) The Child allocators we create in the builder should also be closed when their readers are closed Both of these changes will probably require a bit of refactoring. This would help us with leak detection for direct memory as well as eliminate the Heap memory leak we currently are creating by leaving the child allocators alive forever. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [x] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
