L. C. Hsieh created SPARK-59854:
-----------------------------------
Summary: Report the column in decoding errors of the vectorized
Parquet reader
Key: SPARK-59854
URL: https://issues.apache.org/jira/browse/SPARK-59854
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Most decoding errors thrown by the vectorized Parquet reader while reading a
batch, e.g. the ParquetDecodingExceptions from VectorizedRleValuesReader
(SPARK-59832) and from the PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY
readers (SPARK-59812), do not say which column is corrupt. The error only names
the file, which makes a corrupt page hard to locate in a wide table.
Errors thrown while initializing a page are already wrapped with the page and
the column by VectorizedColumnReader (readPageV1/readPageV2/initDataReader).
Decoding errors thrown later from VectorizedColumnReader.readBatch could be
wrapped the same way, e.g. by catching ParquetDecodingException there and
adding the column descriptor.
This came up in the review of SPARK-59832
(https://github.com/apache/spark/pull/59117).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
