schenksj commented on code in PR #4533:
URL: https://github.com/apache/datafusion-comet/pull/4533#discussion_r3345339988
##########
spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala:
##########
@@ -1095,4 +1095,23 @@ class CometArrayExpressionSuite extends CometTestBase
with AdaptiveSparkPlanHelp
}
}
}
+
+ test("array of structs with nullability-divergent children") {
+ // Spark's type coercion compares element types with `sameType`, which
ignores nullability,
+ // so two struct children that differ ONLY in a nested field's nullability
get no unifying
+ // cast -- CreateArray keeps children of different StructTypes.
DataFusion's make_array asserts
+ // strict element-type equality (down to nested nullability) and panics on
the mismatch. Comet
+ // must decline this CreateArray so Spark's evaluator handles it.
+ withParquetTable((0 until 5).map(i => (i, i.toLong)), "tbl") {
+ val df = spark
+ .table("tbl")
+ .select(
+ array(
+ // ct is NOT NULL (literal)
+ struct(col("_1").as("id"), lit("a").as("ct")),
+ // ct is NULLABLE (when without otherwise) -- same type, different
nullability
+ struct(col("_1").as("id"), when(col("_1") === 0,
lit("b")).as("ct"))).as("arr"))
+ checkSparkAnswer(df)
Review Comment:
Thanks @andygrove, both addressed in `7110a0a3a`.
Switched the existing test to `checkSparkAnswerAndFallbackReason` so it
asserts the specific decline reason (`CreateArray children have mismatched data
types`) instead of only matching the answer.
Added a map test, `array of maps with nullability-divergent struct
values`, which wraps the same nested-nullability divergence in a `MapType`
value. That covers `normalizeContainerNullability`'s
`MapType` branch. The struct field nullability difference survives
container-nullability normalization, so `CreateArray` still declines, mirroring
the struct case. Both tests pass under Spark 4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
