marin-ma opened a new pull request, #13227: URL: https://github.com/apache/gluten/pull/13227
## What changes were proposed in this pull request? When micro-benchmark dumping is enabled (`spark.gluten.sql.benchmark_task.*`), `VeloxWholeStageDumper::dumpInputIterator` creates a dedicated leaf memory pool (`retrieve_iterator.N`) that is owned only by the returned `ParquetStreamReaderIterator`. Vectors read from the dumped file are shared zero-copy into the Velox task by `RowVectorStream::next`, and Velox vectors hold a raw `MemoryPool*`. The reader iterator is destroyed as soon as the value stream is exhausted (and in any case before downstream operators are closed, since `Driver::closeOperators` closes source-first), so vectors still referenced by downstream operators or by batches handed to Java are later freed against a destroyed pool. In release builds `MemoryPoolImpl::~MemoryPoolImpl` only logs `Memory leak (Used memory)`, so the failure surfaces as a SIGSEGV in `AlignedBuffer::freeToPool` during `ColumnarBatchOutIterator.nativeClose`. This PR passes the `VeloxMemoryManager` to `VeloxWholeStageDumper` and uses the Runtime's leaf pool for the Parquet reader. That pool is owned by the memory manager, which is released after the Runtime, so it outlives every iterator and batch created by the Runtime. The Parquet writer keeps using a short-lived aggregate child pool as before. ## How was this patch tested? Added `VeloxWholeStageDumperTest` (`velox_whole_stage_dumper_test`), which dumps and re-reads batches, asserts the vectors are backed by the memory manager's leaf pool, and verifies the batches can be released safely after the reader iterator is destroyed. The last step is the use-after-free with the previous code. Was this patch authored or co-authored using generative AI tooling? Test generated by Claude Fable 5.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
