andygrove commented on code in PR #4533:
URL: https://github.com/apache/datafusion-comet/pull/4533#discussion_r3343689744
##########
spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala:
##########
@@ -1095,4 +1095,23 @@ class CometArrayExpressionSuite extends CometTestBase
with AdaptiveSparkPlanHelp
}
}
}
+
+ test("array of structs with nullability-divergent children") {
+ // Spark's type coercion compares element types with `sameType`, which
ignores nullability,
+ // so two struct children that differ ONLY in a nested field's nullability
get no unifying
+ // cast -- CreateArray keeps children of different StructTypes.
DataFusion's make_array asserts
+ // strict element-type equality (down to nested nullability) and panics on
the mismatch. Comet
+ // must decline this CreateArray so Spark's evaluator handles it.
+ withParquetTable((0 until 5).map(i => (i, i.toLong)), "tbl") {
+ val df = spark
+ .table("tbl")
+ .select(
+ array(
+ // ct is NOT NULL (literal)
+ struct(col("_1").as("id"), lit("a").as("ct")),
+ // ct is NULLABLE (when without otherwise) -- same type, different
nullability
+ struct(col("_1").as("id"), when(col("_1") === 0,
lit("b")).as("ct"))).as("arr"))
+ checkSparkAnswer(df)
Review Comment:
nit: would be better to use `checkSparkAnswerAndFallbackReason`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
