ErikBPF commented on code in PR #5732:
URL: https://github.com/apache/datafusion-comet/pull/5732#discussion_r3967248333
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -845,30 +844,29 @@ case class CometScanRule(session: SparkSession)
} else {
val readSchema = scanExec.scan.readSchema()
- // Identify complex type columns that would trigger accessor
creation failures
- val complexColumns = readSchema
- .filter(field => isComplexType(field.dataType))
+ // A struct inside a list/map does not make the container null
check a struct check.
+ val structColumns = readSchema
+ .filter(field => field.dataType.isInstanceOf[StructType])
Review Comment:
Addressed in 266c54053528f271a1486cb75e6b02bf5ade684c.
`GetArrayStructFields` now uses `containsNull || field.nullable` for both
its declared list element field and runtime list construction, and derives
outer nullability from the input list. The existing parent-null helper still
shares value buffers.
Added the required-child Iceberg regression with `[NULL]`, mixed
null/non-null elements, empty lists and a null list, asserting Spark-equivalent
results plus native scan and projection. A Rust regression covers 16
nullability combinations across `List` and `LargeList`; it reproduces the exact
Arrow 59.3.0 panic on the original implementation and passes with the fix.
Validation: native build and JVM packaging passed. On Spark 4.0 / Iceberg
Java 1.10.0, the full `CometIcebergNativeSuite` and `CometFuzzIcebergSuite`
passed: 112 succeeded, 0 failed, 1 existing Iceberg 1.11-gated cancellation.
Rust regression, Rust formatting and Scala formatting passed. Commands are in
the updated PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
