sigmod commented on a change in pull request #32742:
URL: https://github.com/apache/spark/pull/32742#discussion_r654179121
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEPropagateEmptyRelation.scala
##########
@@ -53,8 +54,11 @@ object AQEPropagateEmptyRelation extends
PropagateEmptyRelationBase {
empty(j)
}
- // TODO we need use transformUpWithPruning instead of transformUp
- def apply(plan: LogicalPlan): LogicalPlan = plan.transformUp {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithPruning(
+ // LOCAL_RELATION and TRUE_OR_FALSE_LITERAL pattern are matched at
`PropagateEmptyRelationBase`
Review comment:
commonApplyFunc?
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEPropagateEmptyRelation.scala
##########
@@ -53,8 +54,11 @@ object AQEPropagateEmptyRelation extends
PropagateEmptyRelationBase {
empty(j)
}
- // TODO we need use transformUpWithPruning instead of transformUp
- def apply(plan: LogicalPlan): LogicalPlan = plan.transformUp {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithPruning(
+ // LOCAL_RELATION and TRUE_OR_FALSE_LITERAL pattern are matched at
`PropagateEmptyRelationBase`
+ // LOGICAL_QUERY_STAGE pattern is matched at `AQEPropagateEmptyRelation`
Review comment:
eliminateSingleColumnNullAwareAntiJoin?
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEPropagateEmptyRelation.scala
##########
@@ -53,8 +54,11 @@ object AQEPropagateEmptyRelation extends
PropagateEmptyRelationBase {
empty(j)
}
- // TODO we need use transformUpWithPruning instead of transformUp
- def apply(plan: LogicalPlan): LogicalPlan = plan.transformUp {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithPruning(
+ // LOCAL_RELATION and TRUE_OR_FALSE_LITERAL pattern are matched at
`PropagateEmptyRelationBase`
+ // LOGICAL_QUERY_STAGE pattern is matched at `AQEPropagateEmptyRelation`
+ // We can not specify ruleId here since the LogicalQueryStage is not
immutable.
+ _.containsAnyPattern(LOGICAL_QUERY_STAGE, LOCAL_RELATION,
TRUE_OR_FALSE_LITERAL)) {
Review comment:
This looks correct, but may not have good pruning power? How about:
t = > t.containsAllPatterns(LOGICAL_QUERY_STAGE, JOIN, AND_OR, NULL_CHECK)
|| t.containsAnyPatterns(LOCAL_RELATION, TRUE_OR_FALSE_LITERAL)
The first condition covers eliminateSingleColumnNullAwareAntiJoin while the
second covers commonApplyFunc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
