vinodkc opened a new pull request, #59717: URL: https://github.com/apache/spark/pull/59717
### What changes were proposed in this pull request? This PR fixes two bugs in the vectorized Parquet reader that are hit when, within one batch, a dictionary-encoded data page follows a PLAIN-encoded data page of the same column chunk. 1. **Wrong results from lazy dictionary decoding** (`VectorizedColumnReader.readBatch`). After reading a dictionary-encoded page, the reader attaches the dictionary to the column vector (lazy decoding) only if no non-dictionary values were added to the batch yet. Since SPARK-34859 this was checked with `startRowId == pageFirstRowIndex`, but `ParquetReadState.resetForNewPage` sets `rowId` to `pageFirstRowIndex`, so the check is always true on the first read of a page. The dictionary was then attached over values that a PLAIN page had already written to the batch, and those rows were decoded from dictionary id slots that were never written. This PR restores the check to `startOffset == 0` (no values in the batch yet), so such pages take the existing eager `decodeDictionaryIds` path. This affects flat and nested columns. 2. **`ArrayIndexOutOfBoundsException` for repeated columns** (`VectorizedRleValuesReader.readValues`). Values are written at `state.valueOffset`, but `values` / `nulls` were reserved with `reserveAdditional(total)`, i.e. relative to their own `elementsAppended`. For a dictionary-encoded page, `values` is the dictionary ids vector, whose `elementsAppended` does not count the values a PLAIN page already wrote to the batch, so the reservation falls short and `putInts` overflows. This PR reserves `state.valueOffset + total` instead. In all the other cases the two are equal, so the behavior there is unchanged. ### Why are the changes needed? The Parquet format allows a column chunk to mix dictionary-encoded and PLAIN data pages in any order. parquet-mr only falls back from dictionary to PLAIN for good, so files written by Spark do not hit this, but other writers do. We hit this with files written by Snowflake Openflow, where a list element column of a shredded variant has the pages `RLE_DICTIONARY -> PLAIN -> RLE_DICTIONARY`. Reading such files: - silently returns wrong values, e.g. `sum(aggregate(tags, 0L, (a, x) -> a + x))` gives `31228268` instead of `31232468`, or - fails with `java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 30504 out of bounds for int[3100]` at `OnHeapColumnVector.putInts` via `VectorizedRleValuesReader.readIntegersRepeated`. Setting `spark.sql.parquet.enableNestedColumnVectorizedReader=false` avoids it only for scans that read nested columns; scans of flat columns stay vectorized and can still return wrong results. ### Does this PR introduce _any_ user-facing change? Yes, it is a bug fix. Reading Parquet files in which a dictionary-encoded page follows a PLAIN page in the same batch now returns correct results instead of wrong values or an `ArrayIndexOutOfBoundsException`. ### How was this patch tested? Added tests to `ParquetVectorizedSuite` that build column chunks with a given sequence of PLAIN and dictionary-encoded pages and read them with batch sizes that span the pages, with on-heap and off-heap vectors: - primitive type, PLAIN page followed by a dictionary page (wrong results before the fix) - nested type, PLAIN page followed by a dictionary page, one element per row (wrong results before the fix) - nested type, PLAIN page followed by a dictionary page, more values than rows (`ArrayIndexOutOfBoundsException` before the fix) - nested string array with dictionary, PLAIN and dictionary pages, mirroring the files above (wrong results before the fix) Also ran `ParquetVectorizedSuite`, `ParquetEncodingSuite`, `ParquetIOSuite`, `ParquetColumnIndexSuite`, `ParquetTypeWideningSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetFileFormatV1Suite`, `ParquetFileFormatV2Suite`, `ParquetV1SchemaPruningSuite` and `ParquetV2SchemaPruningSuite`. ### Was this patch authored or co-authored using generative AI tooling? Yes, Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
