[
https://issues.apache.org/jira/browse/SPARK-60788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60788:
-----------------------------------
Labels: pull-request-available (was: )
> Parquet nested schema pruning can misread two-level LISTs and return values
> for missing field IDs
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-60788
> URL: https://issues.apache.org/jira/browse/SPARK-60788
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 4.3.0, 4.2.1, 4.0.5, 4.1.4
> Reporter: Chao Sun
> Priority: Major
> Labels: pull-request-available
>
> h3. Description
> Spark can misread a two-level Parquet LIST of structs after nested schema
> pruning reduces its element to one field. The clipped physical schema can
> look identical to a three-level LIST of primitives, even though the two
> encodings require different interpretations.
> There is a related wrong-result case when field-ID reads are enabled. The row
> reader can return values from a physical field that has the requested name
> but a different field ID.
> h3. Reproduction
> # Write a two-level Parquet LIST named {{legacy}} whose repeated group
> contains two integer fields: {{sibling}} with field ID 18 and {{wanted}} with
> field ID 17.
> # Write two elements with values {{(10, 11)}} and {{(20, 22)}}.
> # Enable {{spark.sql.optimizer.nestedSchemaPruning.enabled}} and
> {{spark.sql.parquet.fieldId.read.enabled}}.
> # Supply a read schema that requests field ID 17 as {{renamed}} and requests
> {{sibling}} with field ID 19. ID 19 does not exist in the file.
> # Read the following projections separately.
> ||Projection||Expected||Actual on Spark master||
> |{{legacy[1].renamed}}|{{22}}|Fails with a Parquet column type mismatch in
> both the row and vectorized readers.|
> |{{legacy.sibling}}|{{[NULL, NULL]}}|The row reader returns {{[10, 20]}}. The
> vectorized reader returns {{[NULL, NULL]}}.|
> The row reader reads a physical field to distinguish a null parent from a
> present parent. When that field has a different ID from the requested field,
> its values should not appear in the result.
> h3. References
> * [Standalone fix and regression tests, Spark PR
> #59997|https://github.com/apache/spark/pull/59997]
> * [Original review
> report|https://github.com/apache/spark/pull/59972#discussion_r4433280742]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
