sunchao commented on code in PR #5414:
URL: https://github.com/apache/datafusion-comet/pull/5414#discussion_r3878071865
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -44,6 +44,100 @@ class CometExpressionSuite extends CometTestBase with
AdaptiveSparkPlanHelper {
val DIVIDE_BY_ZERO_EXCEPTION_MSG =
"""Division by zero. Use `try_divide` to tolerate divisor being 0 and
return NULL instead"""
+ test("struct comparison declines mismatched-nullability operands containing
an empty struct") {
+ // planner.rs's reconcile_nested_comparison_types casts whichever
comparison operand doesn't
+ // already match the nullability-union of both operand types; a
named_struct literal argument
+ // infers a non-nullable field, so comparing it against a schema-nullable
column of the same
+ // shape casts one side. DataFusion casts a struct field-by-field even
when only that one
+ // field's nullability differs, and casting a zero-field struct to itself
still fails -- see
+ // SupportLevel.containsEmptyStruct.
+ withSQLConf(
+ CometConf.COMET_EXEC_LOCAL_TABLE_SCAN_ENABLED.key -> "true",
+ SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
+ "org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation") {
+ val schema =
+ StructType(Seq(StructField("outer", StructType(Seq(StructField("e",
StructType(Nil)))))))
+ val data =
+ java.util.List.of[Row](Row(Row(Row())), Row(Row(null)), Row(null))
+ val df = spark.createDataFrame(data, schema)
+ df.createOrReplaceTempView("empty_struct_cmp")
+
+ checkSparkAnswerAndFallbackReason(
+ "SELECT outer <=> named_struct('e', struct()) FROM empty_struct_cmp",
+ "on differently-typed operands containing an empty struct is not
supported")
Review Comment:
[P2] Make this fixture exercise the comparison guard
This assertion fails in all six current expression CI jobs: the actual
reason is `Unsupported data type StructType(StructField(e,StructType(),true))`,
rather than the comparison-mismatch reason. The checked Spark 3.5 coercion path
merges nested nullability and folds the named_struct operand into a non-null
struct literal, which Comet rejects earlier. The fixture therefore blocks the
expression test jobs without validating its stated guard. Verify the analyzed
operand types or use a focused serde test with genuinely differing types;
changing only the expected string would stop testing the intended guard.
[Current CI
example](https://github.com/apache/datafusion-comet/actions/runs/33121358952/job/98722616148).
##########
spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala:
##########
@@ -890,6 +891,73 @@ class CometArrayExpressionSuite extends CometTestBase with
AdaptiveSparkPlanHelp
}
}
+ test("array literal of empty struct falls back instead of crashing
planning") {
+ // `array(array(struct()))` constant-folds to a literal whose
(doubly-nested) array element
+ // type is an empty struct. `makeListLiteral` has no case for StructType
at all (empty or
+ // not) -- letting this reach the literal path throws a bare
`scala.MatchError` at plan
+ // time rather than a graceful fallback. See
CometLiteral.isListLiteralElementSupported.
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "test.parquet")
+ spark.range(10).write.parquet(path.toString)
+ withTempView("t1") {
+ spark.read.parquet(path.toString).createOrReplaceTempView("t1")
+ checkSparkAnswerAndFallbackReason(
+ "SELECT id, array(array(struct())) FROM t1",
+ "Unsupported data type array<array<struct")
Review Comment:
[P2] Match the literal fallback's actual type rendering
The fallback helper performs case-sensitive substring matching, but
`CometLiteral.getSupportLevel` interpolates the DataType object and emits
`Unsupported data type ArrayType(ArrayType(StructType(),false),false)`, not the
lowercase catalog syntax asserted here. The fallback and result comparison
succeed, then this new assertion fails in all six current expression CI jobs.
Match the serializer's actual diagnostic contract, or deliberately standardize
that diagnostic and its expectations. [Current CI
example](https://github.com/apache/datafusion-comet/actions/runs/33121358952/job/98722616148).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
