frroliveira opened a new pull request, #59800: URL: https://github.com/apache/spark/pull/59800
### What changes were proposed in this pull request? This tries to ensure that off-heap memory allocated by the Parquet reader is deallocated. The class `ParquetColumnVector` uses the method `WritableColumnVector.reserveNewColumn` in 4 scenarios. Given these column vectors don't belong to ColumnarBatches and the class is not calling `WritableColumnVector.close` when memory mode is OFF_HEAP these column vectors are leaking. To fix, the class `WritableColumnVector` now holds references to these column vectors and closes them on `WritableColumnVector.close`. ### Why are the changes needed? This will prevent OOMs for long running applications. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The 4 scenarios using `WritableColumnVector.reserveNewColumn` are tested in `ParquetVectorizedSuite`. Definition levels and repetition levels are tested by existing tests. 2 new tests were added for the other 2 scenarios. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
