viirya opened a new pull request, #59542:
URL: https://github.com/apache/spark/pull/59542
### What changes were proposed in this pull request?
This is a follow-up of SPARK-59832 (#59117). That change added the page and
the column to errors from initializing a data page in `VectorizedColumnReader`.
This PR adds the column to the decoding errors that still do not report it:
- **Decoding a batch.** `readBatch` wraps a `ParquetDecodingException`
thrown while decoding values or levels in a `ParquetDecodingException` that
names the page and the column: `could not decode page <page> in col <column>:
<original message>`. This covers the RLE, PLAIN, DELTA_LENGTH_BYTE_ARRAY,
DELTA_BYTE_ARRAY and BYTE_STREAM_SPLIT readers. Only `ParquetDecodingException`
is caught, so other exceptions keep their types.
`SchemaColumnConvertNotSupportedException` and `SparkUpgradeException`, which
`FileDataSourceV2.attachFilePath` dispatches on, are not wrapped.
- **Reading a page.** parquet-java decompresses ZSTD pages eagerly, inside
`PageReader.readPage()` and `readDictionaryPage()`, which were outside every
catch. Their `ParquetDecodingException` (`could not decompress page`) now gets
the column. Only `ParquetDecodingException` is caught here as well, because
these calls also decrypt pages, and decryption errors are not decoding errors.
- **Decoding the dictionary.** `Encoding.initDictionary` only parses the
bytes of the dictionary page, so a `RuntimeException` from it comes from the
page or its metadata. It is now wrapped in a `ParquetDecodingException` with
the column. Examples are an `IndexOutOfBoundsException` from a truncated BINARY
dictionary, and an `UnsupportedOperationException` from a dictionary page whose
encoding has no dictionary.
- **DELTA_BINARY_PACKED header.**
`VectorizedDeltaBinaryPackedReader.initFromPage` checked the page value count
and the mini block size with `checkArgument`, which throws an
`IllegalArgumentException`. These checks now throw a
`ParquetDecodingException`, like the block size checks next to them, so the
existing page wrapper adds the page and the column.
Each wrapper keeps the original message and adds a single layer to the cause
chain.
### Why are the changes needed?
Without the column, a decoding error from a corrupted file does not say
which column is affected. For example, a truncated RLE run fails with only
`Corrupted RLE data: reading past the end of the encoded values`. SNAPPY and
GZIP pages already report the column when decompression fails, because they are
decompressed lazily inside the existing catch. ZSTD pages did not report it.
### Does this PR introduce _any_ user-facing change?
Yes, in error messages only. The decoding errors above now name the column,
and the page for errors while decoding a batch. A corrupt DELTA_BINARY_PACKED
header and an undecodable dictionary now fail with a `ParquetDecodingException`
that names the column. Before, they failed with an `IllegalArgumentException`,
`IndexOutOfBoundsException` or `UnsupportedOperationException`, and the
original exception is kept as the cause. Valid files are not affected.
### How was this patch tested?
New tests:
- `ParquetVectorizedSuite`:
- A truncated RLE dictionary id section and truncated PLAIN values report
the page and the column, with the original `ParquetDecodingException` as the
cause.
- A truncated BINARY dictionary and a dictionary page with a
non-dictionary encoding report the column.
- An invalid DELTA_BINARY_PACKED header reports the column, and a page
with no values fails with a `ParquetDecodingException`.
- `ParquetIOSuite`: a ZSTD data page and a ZSTD dictionary page that cannot
be decompressed report the column once.
All new tests fail without this change. Existing suites also pass:
`VectorizedRleValuesReaderSuite`, `ParquetEncodingSuite`,
`ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`,
`ParquetV1QuerySuite`, `ParquetV2QuerySuite`,
`ParquetDeltaByteArrayEncodingSuite`,
`ParquetDeltaLengthByteArrayEncodingSuite`, `VectorizedPlainValuesReaderSuite`,
`ParquetVectorUpdaterSuite` and `ParquetCompressionCodecPrecedenceSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
