[
https://issues.apache.org/jira/browse/SPARK-60490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod KC updated SPARK-60490:
-----------------------------
Labels: correctness pull-request-available (was: pull-request-available)
> Fix vectorized Parquet reader when a PLAIN page is followed by a dictionary
> page in a batch
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-60490
> URL: https://issues.apache.org/jira/browse/SPARK-60490
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.0, 4.1.0, 4.0.0, 4.2.0, 4.3.0, 5.0.0
> Reporter: Vinod KC
> Priority: Major
> Labels: correctness, pull-request-available
>
> The vectorized Parquet reader can return wrong results or fail with an
> {{ArrayIndexOutOfBoundsException}} when, within one batch, a
> dictionary-encoded data page follows a PLAIN-encoded data page of the same
> column chunk.
> The Parquet format allows a column chunk to mix dictionary-encoded and PLAIN
> data pages in any order. parquet-mr's fallback from dictionary to PLAIN is
> permanent for the column chunk, so files written by Spark don't hit this, but
> other writers do.
> For example, files written by Snowflake Openflow contain a list element
> column of a shredded variant with the page sequence {{{}RLE_DICTIONARY ->
> PLAIN -> RLE_DICTIONARY{}}}.
> *Symptoms*
> * Silently wrong values, e.g. {{sum(aggregate(tags, 0L, (a, x) -> a + x))}}
> returns {{31228268}} instead of {{{}31232468{}}}.
> * For repeated (array/map) columns, a failure such as:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index
> 30504 out of bounds for int[3100]
> at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.putInts
> at
> org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegersRepeated
> ...
> {noformat}
> With off-heap column vectors
> ({{{}spark.sql.columnVector.offheap.enabled=true{}}}),
> {{OffHeapColumnVector.putInts}} has no bounds check, so this overflow writes
> past the allocated native memory instead of throwing.
> Setting {{spark.sql.parquet.enableNestedColumnVectorizedReader=false}} avoids
> the problem only for scans that read nested columns. Scans of only flat
> columns stay vectorized and can still return wrong results.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
