Vinod KC created SPARK-60490:
--------------------------------
Summary: Fix vectorized Parquet reader when a PLAIN page is
followed by a dictionary page in a batch
Key: SPARK-60490
URL: https://issues.apache.org/jira/browse/SPARK-60490
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0, 4.0.0, 4.1.0, 3.5.0, 4.3.0, 5.0.0
Reporter: Vinod KC
The vectorized Parquet reader can return wrong results or fail with an
{{ArrayIndexOutOfBoundsException}} when, within one batch, a dictionary-encoded
data page follows a PLAIN-encoded data page of the same column chunk.
The Parquet format allows a column chunk to mix dictionary-encoded and PLAIN
data pages in any order. parquet-mr's fallback from dictionary to PLAIN is
permanent for the column chunk, so files written by Spark don't hit this, but
other writers do.
For example, files written by Snowflake Openflow contain a list element column
of a shredded variant with the page sequence {{{}RLE_DICTIONARY -> PLAIN ->
RLE_DICTIONARY{}}}.
*Symptoms*
* Silently wrong values, e.g. {{sum(aggregate(tags, 0L, (a, x) -> a + x))}}
returns {{31228268}} instead of {{{}31232468{}}}.
* For repeated (array/map) columns, a failure such as:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index
30504 out of bounds for int[3100]
at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.putInts
at
org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegersRepeated
...
{noformat}
With off-heap column vectors
({{{}spark.sql.columnVector.offheap.enabled=true{}}}),
{{OffHeapColumnVector.putInts}} has no bounds check, so this overflow writes
past the allocated native memory instead of throwing.
Setting {{spark.sql.parquet.enableNestedColumnVectorizedReader=false}} avoids
the problem only for scans that read nested columns. Scans of only flat columns
stay vectorized and can still return wrong results.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
