Chao Sun created SPARK-60788:
--------------------------------
Summary: Parquet nested schema pruning can misread two-level LISTs
and return values for missing field IDs
Key: SPARK-60788
URL: https://issues.apache.org/jira/browse/SPARK-60788
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Chao Sun
h3. Description
Spark can misread a two-level Parquet LIST of structs after nested schema
pruning reduces its element to one field. The clipped physical schema can look
identical to a three-level LIST of primitives, even though the two encodings
require different interpretations.
There is a related wrong-result case when field-ID reads are enabled. The row
reader can return values from a physical field that has the requested name but
a different field ID.
h3. Reproduction
# Write a two-level Parquet LIST named {{legacy}} whose repeated group contains
two integer fields: {{sibling}} with field ID 18 and {{wanted}} with field ID
17.
# Write two elements with values {{(10, 11)}} and {{(20, 22)}}.
# Enable {{spark.sql.optimizer.nestedSchemaPruning.enabled}} and
{{spark.sql.parquet.fieldId.read.enabled}}.
# Supply a read schema that requests field ID 17 as {{renamed}} and requests
{{sibling}} with field ID 19. ID 19 does not exist in the file.
# Read the following projections separately.
||Projection||Expected||Actual on Spark master||
|{{legacy[1].renamed}}|{{22}}|Fails with a Parquet column type mismatch in both
the row and vectorized readers.|
|{{legacy.sibling}}|{{[NULL, NULL]}}|The row reader returns {{[10, 20]}}. The
vectorized reader returns {{[NULL, NULL]}}.|
The row reader reads a physical field to distinguish a null parent from a
present parent. When that field has a different ID from the requested field,
its values should not appear in the result.
h3. References
* [Standalone fix and regression tests, Spark PR
#59997|https://github.com/apache/spark/pull/59997]
* [Original review
report|https://github.com/apache/spark/pull/59972#discussion_r4433280742]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
