L. C. Hsieh created SPARK-60317:
-----------------------------------
Summary: Report out-of-range dictionary ids in the vectorized
Parquet reader
Key: SPARK-60317
URL: https://issues.apache.org/jira/browse/SPARK-60317
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: L. C. Hsieh
A dictionary id at or beyond the size of the dictionary fails with an
{{ArrayIndexOutOfBoundsException}} from {{Dictionary.decodeTo*}}. With eager
decoding it is thrown in {{VectorizedColumnReader.readBatch}} without the page
or the column. With lazy dictionary decoding {{readBatch}} succeeds, and the
exception is thrown later from {{ParquetDictionary}} in the consumer, e.g.
{{Index 5 out of bounds for length 2}} for a 2-entry INT32 dictionary. That is
outside the error handling of {{FileScanRDD}}, so the error names neither the
file nor the column, and {{spark.sql.files.ignoreCorruptFiles}} cannot skip it.
The id reader could check the ids against the dictionary size and throw a
{{ParquetDecodingException}} with the column. An id is always below 2^bitWidth,
so the check is only needed when the dictionary is smaller than that: once per
RLE run, and once per group of 8 values for bit-packed runs. Since this is on
the hot path, the change needs a benchmark.
Found in the review of SPARK-59854 (https://github.com/apache/spark/pull/59542).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
