ErikBPF commented on code in PR #5732:
URL: https://github.com/apache/datafusion-comet/pull/5732#discussion_r3944260254
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -845,13 +844,16 @@ case class CometScanRule(session: SparkSession)
} else {
val readSchema = scanExec.scan.readSchema()
- // Identify complex type columns that would trigger accessor
creation failures
+ // IS NULL/NOT NULL on struct columns must fall back:
iceberg-rust's Arrow
+ // predicate visitor cannot project struct columns (see
project_column in
+ // iceberg-rust arrow/reader/predicate_visitor.rs). List and map
columns
+ // evaluate through arrow's native is_null/is_not_null and are
supported.
val complexColumns = readSchema
- .filter(field => isComplexType(field.dataType))
+ .filter(field => field.dataType.isInstanceOf[StructType])
Review Comment:
Done — converted the five affected assertions (array IS NULL, array-element
filter, whole-array equality, map IS NULL, map-key access) to
`checkIcebergNativeScan` in 37509cba1, so they now assert the scan stays native
while the post-scan Comet filter enforces the predicate. Struct IS NULL and
whole-struct equality remain as fallback cases with updated reasons
(struct-only gate; Iceberg Java does not push whole-struct equality).
`CometIcebergNativeSuite`: 99 succeeded, 0 failed (1 pre-existing version-gated
cancel).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
