yeshengm commented on issue #25164: [SPARK-28375][SQL] Prevent the PullupCorrelatedPredicates optimizer rule from removing predicates if run multiple times URL: https://github.com/apache/spark/pull/25164#issuecomment-512525515 Yep. It does work! On Wed, Jul 17, 2019 at 11:57 AM Dilip Biswal <[email protected]> wrote: > *@dilipbiswal* commented on this pull request. > ------------------------------ > > In > sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala > <https://github.com/apache/spark/pull/25164#discussion_r304591303>: > > > @@ -275,13 +275,16 @@ object PullupCorrelatedPredicates extends Rule[LogicalPlan] with PredicateHelper > plan transformExpressions { > case ScalarSubquery(sub, children, exprId) if children.nonEmpty => > val (newPlan, newCond) = pullOutCorrelatedPredicates(sub, outerPlans) > - ScalarSubquery(newPlan, newCond, exprId) > + val conds = newCond ++ children.filter(_.isInstanceOf[Predicate]) > > @yeshengm <https://github.com/yeshengm> Ok.. sounds good. > > One question, can we not make this PullupCorrelatedPredicates idempotent > now the way it is (i.e when these two rules are separate) ? If we did > something like this : > > case l @ ListQuery(sub, _, exprId, childOutputs) => > val (newPlan, newCond) = pullOutCorrelatedPredicates(sub, outerPlans) > if (newCond.isEmpty) { > // Perhaps just returning `l` may work as well. But in case we r relying on > // the de-dup processing somehow.. > ListQuery(newPlan, l.children, exprId, childOutputs) > } else { > ListQuery(newPlan, newCond, exprId, childOutputs) > } > > will it work ? Or you tried it already ? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/25164?email_source=notifications&email_token=AC5TTEM4YUV5AB4Y2GUQ4DDP75TSFA5CNFSM4IDZAVE2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB6YOUFY#discussion_r304591303>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AC5TTEPCLRIBBWFABPQSURLP75TSFANCNFSM4IDZAVEQ> > . > -- Yesheng
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
