frroliveira opened a new pull request, #59329:
URL: https://github.com/apache/spark/pull/59329
### What changes were proposed in this pull request?
This tries to ensure that off-heap memory allocated by the Parquet reader is
deallocated. The class `ParquetColumnVector` uses the method
`WritableColumnVector.reserveNewColumn` in 4 scenarios:
- Repetition levels
- Definition levels
- Dummy vector to handle missing structs
- Variant type
Given these column vectors don't belong to `ColumnarBatch`es and the class
is not calling `WritableColumnVector.close` when memory mode is OFF_HEAP these
column vectors are leaking.
To fix the method `VectorizedParquetRecordReader.close` calls the new method
in `ParquetColumnVector.close`. This last one mimics the logic in the
constructor of `ParquetColumnVector` to deallocate memory of the 4 scenarios
identified above.
### Why are the changes needed?
This will prevent OOMs for long running applications.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Given there is no way to track/debug memory allocations in
`OffHeapColumnVector` the change was tested manually:
- Added println to invocations of `WritableColumnVector.reserveNewColumn` in
`ParquetColumnVector`
- Added println to invocations of `WritableColumnVector.close` in
`ParquetColumnVector`
- Run the suites
- org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite
- org.apache.spark.sql.VariantSuite
- Ensured that the number of occurrences of each scenario matched correctly
The rest of the test suite was able to catch issues (fixed now)
-
org.apache.spark.sql.execution.datasources.PushVariantIntoScanV2VectorizedSuite
- org.apache.spark.sql.execution.WholeStageCodegenSuite
-
org.apache.spark.sql.execution.datasources.parquet.ParquetStorageFilterSuite
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
