sunchao commented on code in PR #5732:
URL: https://github.com/apache/datafusion-comet/pull/5732#discussion_r3942777912
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -845,13 +844,16 @@ case class CometScanRule(session: SparkSession)
} else {
val readSchema = scanExec.scan.readSchema()
- // Identify complex type columns that would trigger accessor
creation failures
+ // IS NULL/NOT NULL on struct columns must fall back:
iceberg-rust's Arrow
+ // predicate visitor cannot project struct columns (see
project_column in
+ // iceberg-rust arrow/reader/predicate_visitor.rs). List and map
columns
+ // evaluate through arrow's native is_null/is_not_null and are
supported.
val complexColumns = readSchema
- .filter(field => isComplexType(field.dataType))
+ .filter(field => field.dataType.isInstanceOf[StructType])
Review Comment:
### Correctness
[P2] Update the existing Iceberg fallback tests
Could you update the affected cases in `CometIcebergNativeSuite` along with
this gate? The existing array/map `IS NULL` tests still call
`checkIcebergNativeScanFallback` (lines 2358 and 2469), whose helper explicitly
asserts that no `CometIcebergNativeScanExec` exists. Those assertions now
reject the native scans this change enables. The array-element, whole-array
equality, and map-key cases also retain fallback assertions based on the
implicit `NOT NULL` restriction being removed here. This suite runs alongside
`CometFuzzIcebergSuite` in the reader CI group, so adding the new test does not
replace the conflicting expectations. Please convert the affected cases to
native-scan assertions while retaining their Spark-result comparisons, and run
that suite. This is a source-verified assertion conflict, not an observed CI
failure: the current workflows are still `action_required`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
