Chao Sun created SPARK-60789:
--------------------------------
Summary: Nested Dataset projections can return a sibling field or
fail under case-insensitive analysis
Key: SPARK-60789
URL: https://issues.apache.org/jira/browse/SPARK-60789
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Chao Sun
h3. Description
A typed Dataset serializer can retain a wider struct than a nested projection
requests. If Spark binds the projection using field positions from the narrower
struct, the query can silently return a sibling's value. The same optimizer
rule can also fail to resolve a case-insensitive reference to a serializer
output.
h3. Wrong-result reproduction
Run in spark-shell:
{code:scala}
import spark.implicits._
val ds = spark.range(1).map { _ =>
Tuple1((Map("k" -> ("map-a", "map-b")), ("other-a", "other-b")))
}
ds.selectExpr("_1._1['k']._2", "_1._2._1").show(false)
{code}
*Expected:* {{(map-b, other-a)}}
*Actual:* {{(map-a, other-a)}}
The Map key is constant. Selecting the sibling expression alongside the Map
expression is significant because the serializer can retain its original field
layout.
h3. Case-insensitive reproduction
With {{spark.sql.caseSensitive=false}}, use a typed Dataset with an output
column named {{Items}} containing an array of tuples. Select {{items[0]._2}}
and {{ITEMS._2}}. On Spark master, this can throw
{{ArrayIndexOutOfBoundsException}} during optimization.
Both references should resolve to {{Items}}. Spark should bind the nested
access using the schema the serializer actually emits and preserve the output
names requested by the query.
h3. References
* [Standalone fix and both regression tests, Spark PR
#59996|https://github.com/apache/spark/pull/59996]
* [Original review
report|https://github.com/apache/spark/pull/59972#discussion_r4433280743]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
