cloud-fan commented on PR #49202: URL: https://github.com/apache/spark/pull/49202#issuecomment-2567333471
This LGTM overall, my last suggestion is to narrow the scope of the `With` expression generated during filter pushdown. We should rewrite the `With` expression immediately at the end of the filter pushdown rule. However, this causes conflicts as we need to push filters through the `Project` generated by `With` in the next iteration, and we need to have a way to make it idempotent. My idea is, when rewriting `With` inside `Filter`, and the generated `Project` is the same as the `Project` above `Filter`, we remove the above `Project`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
