viirya opened a new pull request, #59483: URL: https://github.com/apache/spark/pull/59483
### What changes were proposed in this pull request? Backport of #59117 to branch-4.1. This branch does not have `VectorizedRleValuesReaderSuite`, `ParquetTestAccess` or `VectorizedRleValuesReaderTestUtils`, so this PR adds a `VectorizedRleValuesReaderSuite` with the 14 tests from #59117 and the helpers they use. It creates and drives the package-private `ParquetReadState` directly instead of through the `ParquetTestAccess` bridge. It also does not have SPARK-56895, so `readGroup` still slices a bit-packed run one group at a time; the new checks run before that loop, as on master. Tests that came into the conflicts as context but are not on this branch are left out (the SPARK-57828 test in `ParquetIOSuite` and the SPARK-59831 test in `ParquetVectorizedSuite`), and `ParquetVectorizedSuite` now imports `BytesInput`, `Encoding` and `Statistics` for the new dictionary page test. ### Why are the changes needed? Same as #59117. On a corrupt page, the vectorized RLE reader, which decodes definition/repetition levels, dictionary ids and RLE booleans, silently returned wrong values or looped forever when the encoded data ended early, and a corrupt bit-packed header could overflow `currentCount` or allocate a huge buffer. The reader now fails with a `ParquetDecodingException`. ### Does this PR introduce _any_ user-facing change? Only for corrupt or non-compliant Parquet files, as described in #59117: such pages now fail with a `ParquetDecodingException` instead of returning wrong values or hanging, and with `spark.sql.files.ignoreCorruptFiles=true` the rest of such a file is skipped. A few of these pages could be read before and gave the right result by chance; #59117 lists them. ### How was this patch tested? The 18 tests added in #59117 pass on this branch, as do the existing tests in `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetDeltaByteArrayEncodingSuite`, `ParquetDeltaLengthByteArrayEncodingSuite` and `VectorizedPlainValuesReaderSuite`. Without the fix, the 13 negative tests in `VectorizedRleValuesReaderSuite` fail, as on master. ``` build/sbt 'sql/testOnly *VectorizedRleValuesReaderSuite *ParquetEncodingSuite *ParquetVectorizedSuite *ParquetIOSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
