parthchandra commented on code in PR #2948:
URL: https://github.com/apache/datafusion-comet/pull/2948#discussion_r2662264601
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -478,29 +478,33 @@ case class CometScanRule(session: SparkSession) extends
Rule[SparkPlan] with Com
false
}
- // Check for unsupported transform functions in residual expressions
- // iceberg-rust can only handle identity transforms in residuals; all
other transforms
- // (truncate, bucket, year, month, day, hour) must fall back to Spark
+ // Check for transform functions in residual expressions
+ // Non-identity transforms (truncate, bucket, year, month, day, hour)
in residuals
+ // are now supported - they skip row-group filtering and are handled
post-scan by CometFilter.
+ // This is less optimal than row-group filtering but still allows
native execution.
val transformFunctionsSupported =
try {
IcebergReflection.findNonIdentityTransformInResiduals(metadata.tasks) match {
case Some(transformType) =>
- // Found unsupported transform
- fallbackReasons +=
- s"Iceberg transform function '$transformType' in residual
expression " +
- "is not yet supported by iceberg-rust. " +
- "Only identity transforms are supported."
- false
+ // Found non-identity transform - log info and continue with
native scan
+ // Row-group filtering will skip these predicates, but
post-scan filtering will apply
+ logInfo(
+ s"Iceberg residual contains transform '$transformType' - " +
+ "row-group filtering will skip this predicate, " +
+ "post-scan filtering by CometFilter will apply instead.")
+ true // Allow native execution
Review Comment:
Is there a unit test to ensure that the plan does in fact include a native
filter after this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
