SEPURI-SAI-KRISHNA opened a new pull request, #59572:
URL: https://github.com/apache/spark/pull/59572
### What changes were proposed in this pull request?
`VectorizedDeltaByteArrayReader.readValues` and `skipBinary` copied each
value's suffix with
```java
arrayData.appendBytes(suffixLength, suffix.array(), suffix.position());
```
which ignores `suffix.arrayOffset()`. This PR adds it, as the other
vectorized decoders already do (`VectorizedPlainValuesReader`,
`VectorizedDeltaBinaryPackedReader`, and
`ByteBufferOutputWriter.writeArrayByteBuffer`, which
`VectorizedDeltaLengthByteArrayReader` uses).
The open #55924 (SPARK-56897) rewrites this decoder for performance and
reads the suffix through the stream, which would avoid the problem too. This PR
keeps to the two lines, so that it can be backported, and adds a test that
covers either implementation.
### Why are the changes needed?
The suffix is a view of the page's buffer, and its position is relative to
the start of that buffer, not to the start of the array behind it. The two only
coincide when the buffer starts at offset 0 of its array.
With Hadoop vectored IO, which Parquet enables by default since 1.16.0
(apache/parquet-java GH-3156), a column chunk is read into a slice of a larger
buffer. The pages of an uncompressed column chunk are then views at a non-zero
array offset, so the reader copied the bytes from the wrong place and returned
wrong values without an error. A compressed page is decompressed into a buffer
of its own, which is why only uncompressed files are affected.
On Spark 4.2.0 and 4.1.3, with default settings:
```sql
SET spark.sql.parquet.compression.codec=uncompressed;
CREATE TABLE t USING parquet
OPTIONS ('parquet.writer.version' = 'PARQUET_2_0',
'parquet.enable.dictionary' = 'false')
AS SELECT id, CAST(id AS STRING) AS s, CAST(id AS DECIMAL(38, 2)) AS d
FROM range(5);
SELECT * FROM t ORDER BY id;
```
```
0 \u0000 0.00
1 \u0000 0.00
2 \u0000 0.00
3 \u0000 0.00
4 0 1.00
```
With `spark.sql.parquet.enableVectorizedReader=false`, or with
`parquet.hadoop.vectored.io.enabled=false`, the same file reads as `0 0 0.00`,
`1 1 1.00`, ..., `4 4 4.00`. A wrong value can also make a later step fail,
e.g. with `CAST_INVALID_INPUT` on a join key or `NUMERIC_VALUE_OUT_OF_RANGE` on
a decimal.
`DELTA_BYTE_ARRAY` is what the Parquet v2 writer uses for `BINARY` and
`FIXED_LEN_BYTE_ARRAY` columns without a dictionary, or once a dictionary falls
back. Spark 4.0.x uses Parquet 1.15.2, where vectored IO is off by default, so
it is affected only with `parquet.hadoop.vectored.io.enabled=true`.
### Does this PR introduce _any_ user-facing change?
Yes. The vectorized Parquet reader now returns the right values for
`DELTA_BYTE_ARRAY` columns in uncompressed files, where it returned wrong
values in 4.1.x and 4.2.x.
### How was this patch tested?
A new test in `ParquetEncodingSuite` writes an uncompressed v2 file with
dictionary encoding off, checks in the footer that the `STRING` and
`DECIMAL(38, 2)` columns are `DELTA_BYTE_ARRAY` and uncompressed, and reads it
with vectored IO on, on heap and off heap. It reads the whole file, which goes
through `readValues`, and a filter on a column with smaller pages, which skips
rows inside the pages of the other columns and goes through `skipBinary`. It
fails without this change. It also fails with only the `readValues` line fixed,
so both lines are covered.
`ParquetEncodingSuite`, `ParquetDeltaByteArrayEncodingSuite`,
`ParquetDeltaLengthByteArrayEncodingSuite` and `ParquetDeltaEncodingSuite` pass.
The repro above was also run on the official 4.2.0, 4.1.3 and 4.0.4
binaries: 4.2.0 and 4.1.3 return the wrong values, and 4.0.4 returns them only
with vectored IO turned on.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
