szehon-ho commented on code in PR #58145:
URL: https://github.com/apache/spark/pull/58145#discussion_r4151291161
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:
##########
@@ -1019,11 +1049,18 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan]
with PredicateHelper {
// merge unsound. See DataSourceV2ScanRelation.pushedFilters.
val scanRelation = DataSourceV2ScanRelation(sHolder.relation,
wrappedScan, output,
pushedFilters = sHolder.pushedFilterExpressions,
+ inferredFilters = remappedInferredFilters,
// The one site that grants mergeability: a plain scan carrying only
reproducible pushdowns
// (column pruning + deterministic filters) may be fused. See
hasBlockingPushdown.
mergeableScan = !hasBlockingPushdown(sHolder))
- val finalFilters = normalizedFilters.map(projectionFunc)
+ val inferredAdjustmentFilters =
Review Comment:
One clarification on the scope of the refactoring: the minimum-estimate path
requires a `DataSourceV2ScanRelation` with nonempty inferred filters,
`useInferredFilterEstimation() == true`, `reflectsFullyPushedDownFilters() ==
false`, and CBO enabled. Outside that path, `BasicStatsPlanVisitor` uses the
existing `FilterEstimation(p).estimate` call.
`childIsLeaf` preserves the existing null-estimation guard after separating
the estimator from the plan node. The ordinary `FilterEstimation(plan)` helper
still passes `plan.child.isInstanceOf[LeafNode]`. The direct scan caller passes
`true` because both predicate groups are estimated from unadjusted leaf scan
statistics. This preserves the existing `IsNull`/`IsNotNull` restriction for
intermediate operators, whose `nullCount` may be inaccurate, while allowing
estimation without constructing temporary `Filter` nodes.
Classic V1 retains its existing calculation. V2 scans that leave the opt-in
false also use the existing estimator; their reported inferred predicates can
still become ordinary executable `Filter` nodes when
`reflectsFullyPushedDownFilters()` is false. Those predicates can affect
statistics through the ordinary estimator, so disabling the opt-in preserves
that behavior rather than disabling inferred-filter handling altogether. A
`V1ScanWrapper` within V2 planning follows the same opt-in conditions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
