[
https://issues.apache.org/jira/browse/SPARK-60791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-60791:
-----------------------------
Affects Version/s: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
(was: 4.2.0)
> ORC V2 returns the wrong nested field when pruning under positional schema
> evolution
> ------------------------------------------------------------------------------------
>
> Key: SPARK-60791
> URL: https://issues.apache.org/jira/browse/SPARK-60791
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
> Reporter: Chao Sun
> Priority: Major
>
> h3. Description
> With positional schema evolution enabled, pruning an array element to its
> second struct field can cause the ORC V2 reader to read the first physical
> field instead. The query succeeds and silently returns the wrong value.
> h3. Reproduction
> Run in spark-shell:
> {code:scala}
> spark.conf.set("spark.sql.sources.useV1SourceList", "")
> spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", "true")
> val path =
> java.nio.file.Files.createTempDirectory("orc-positional").resolve("data").toString
> spark.sql("""
> SELECT array(named_struct('sibling', 11, 'wanted', 22)) AS arr,
> 0 AS idx,
> array(named_struct('keep', 33, 'skip', 44)) AS other
> """).write.orc(path)
> val df = spark.read.option("orc.force.positional.evolution", "true").orc(path)
> df.selectExpr("arr[0].wanted", "other[0].keep").show()
> df.selectExpr("arr[idx].wanted", "other[0].keep").show()
> {code}
> ||Projection||Expected||Actual on Spark master||
> |Constant index, {{arr[0].wanted}}|{{(22, 33)}}|{{(11, 33)}}|
> |Index column containing zero, {{arr[idx].wanted}}|{{(22, 33)}}|{{(22, 33)}}|
> Both row and vectorized readers reproduce the wrong result. The value {{11}}
> belongs to {{sibling}}, the first physical child. Pruning should preserve the
> physical position of {{wanted}}.
> h3. References
> * [Review discussion and
> reproduction|https://github.com/apache/spark/pull/59972#discussion_r4433555096]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
