[
https://issues.apache.org/jira/browse/SPARK-60789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-60789:
-----------------------------
Affects Version/s: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
(was: 4.2.0)
> Nested Dataset projections can return a sibling field or fail under
> case-insensitive analysis
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-60789
> URL: https://issues.apache.org/jira/browse/SPARK-60789
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
> Reporter: Chao Sun
> Priority: Major
>
> h3. Description
> A typed Dataset serializer can retain a wider struct than a nested projection
> requests. If Spark binds the projection using field positions from the
> narrower struct, the query can silently return a sibling's value. The same
> optimizer rule can also fail to resolve a case-insensitive reference to a
> serializer output.
> h3. Wrong-result reproduction
> Run in spark-shell:
> {code:scala}
> import spark.implicits._
> val ds = spark.range(1).map { _ =>
> Tuple1((Map("k" -> ("map-a", "map-b")), ("other-a", "other-b")))
> }
> ds.selectExpr("_1._1['k']._2", "_1._2._1").show(false)
> {code}
> *Expected:* {{(map-b, other-a)}}
> *Actual:* {{(map-a, other-a)}}
> The Map key is constant. Selecting the sibling expression alongside the Map
> expression is significant because the serializer can retain its original
> field layout.
> h3. Case-insensitive reproduction
> With {{spark.sql.caseSensitive=false}}, use a typed Dataset with an output
> column named {{Items}} containing an array of tuples. Select {{items[0]._2}}
> and {{ITEMS._2}}. On Spark master, this can throw
> {{ArrayIndexOutOfBoundsException}} during optimization.
> Both references should resolve to {{Items}}. Spark should bind the nested
> access using the schema the serializer actually emits and preserve the output
> names requested by the query.
> h3. References
> * [Standalone fix and both regression tests, Spark PR
> #59996|https://github.com/apache/spark/pull/59996]
> * [Original review
> report|https://github.com/apache/spark/pull/59972#discussion_r4433280743]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
