sunchao commented on code in PR #56575:
URL: https://github.com/apache/spark/pull/56575#discussion_r3457228643
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala:
##########
@@ -376,6 +376,11 @@ class V2ExpressionBuilder(e: Expression, isPredicate:
Boolean = false) extends L
} else {
None
}
+ // A surviving `RuntimeReplaceable` (`eagerReplace = false`) that no
explicit case above pushes
+ // natively is a Spark-internal optimizer node the connector cannot
understand. Fall back to its
+ // concrete `replacement` so the lowered form can still be pushed -- same
boundary rationale as
+ // `DataSourceStrategy.translateLeafNodeFilter` (V1) and
`CachedBatchSerializer.buildFilter`.
+ case r: RuntimeReplaceable => generateExpression(r.replacement,
isPredicate)
Review Comment:
[P2] Preserve the mapping for compound replacement predicates
`translateFilterV2WithMapping` treats the surviving wrapper as its
leaf/`other` case, but this fallback can return a structural `V2And`, `V2Or`,
or `V2Not`. The map then contains only `compoundPredicate -> originalWrapper`.
If a `SupportsPushDownV2Filters` source returns that predicate unchanged
because it cannot fully push it (as the interface contract requires),
`rebuildExpressionFromFilter` matches the compound node first and recursively
visits children that were created inside `V2ExpressionBuilder` and have no map
entries. Planning then throws `Failed to rebuild Expression for filter` instead
of retaining a post-scan filter. For example, this occurs for an `eagerReplace
= false` wrapper whose replacement is `And(a > 1, b < 2)` on a rejecting V2
source.
Could we either unfold before entering the mapping-aware recursion, or make
rebuilding prefer an exact map entry before descending, and add a
rejecting-source regression test? The new test exercises only a leaf
`GreaterThan` translation, so it does not cover this reconstruction path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
