zml1206 commented on PR #49202: URL: https://github.com/apache/spark/pull/49202#issuecomment-2553285752
> I'm very confused. When pushing predicates through Project, the problem we hit is we need to rewrite the attributes in the predicate with the corresponding expression in the Project. If the attribute appears more than once in the predicate, and the expression from Project is expensive, filter pushdown may make the query slower. > > The solution is to wrap the predicate with `With` expression, adding the expression from Project as the common expression, and replace attributes in predicate with the common expression reference. But this PR does some rewriting and I don't understand what it does. Filters support partial pushdown. If an attribute exists in two conditions, one condition is pushed to one level, and the other condition is pushed down to multiple levels. Therefore, after splitting by and, each filter need generates a With, then they need to share commonexprdef. Only the lowest commonexprdef will generate a common expression project. In addition, the child of the original common expression Alias also needs to be replaced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
