dongjoon-hyun opened a new pull request, #59101:
URL: https://github.com/apache/spark/pull/59101
### What changes were proposed in this pull request?
This PR aims to implement `skipFixedLenByteArray` in
`VectorizedDeltaByteArrayReader` by delegating to `skipBinary`, because
`DELTA_BYTE_ARRAY` encodes `FIXED_LEN_BYTE_ARRAY` values in the same way as
`BYTE_ARRAY` values.
### Why are the changes needed?
The Parquet v2 writer encodes `FIXED_LEN_BYTE_ARRAY` columns such as
`DECIMAL(38, 18)` with `DELTA_BYTE_ARRAY` when dictionary encoding is disabled.
When Parquet column index filtering skips rows of such a column, the vectorized
reader fails because `VectorizedDeltaByteArrayReader` does not implement
`skipFixedLenByteArray`.
```scala
spark.range(0, 2000).selectExpr("id as _1", "cast(id as decimal(38, 18)) as
_2")
.coalesce(1)
.write
.option("parquet.writer.version", "PARQUET_2_0")
.option("parquet.enable.dictionary", "false")
.option("parquet.page.size", "2048")
.option("parquet.page.row.count.limit", "500")
.parquet("t")
spark.read.parquet("t").filter("_1 >= 500 and _1 < 1000").collect()
```
```
org.apache.spark.SparkUnsupportedOperationException:
[UNSUPPORTED_CALL.WITHOUT_SUGGESTION] Cannot call the method
"skipFixedLenByteArray" of the class
"org.apache.spark.sql.execution.datasources.parquet.VectorizedReaderBase".
SQLSTATE: 0A000
```
### Does this PR introduce _any_ user-facing change?
Yes, this fixes a bug that has existed since Apache Spark
[v3.3.0](https://github.com/apache/spark/releases/tag/v3.3.0) (2022-06-09).
### How was this patch tested?
Pass the CIs with the newly added test cases, which fail without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
