maytasm opened a new pull request, #43471: URL: https://github.com/apache/spark/pull/43471
### What changes were proposed in this pull request? This PR adds a new feature(which is disabled by default to maintain current behavior) that would evaluate scalar subqueries in the Optimizer before rule to push down filter. Note that this PR is similar (in intent) to: - https://github.com/apache/spark/pull/23802 - https://github.com/apache/spark/pull/41088 but is not the same. This is because the above two PRs requires the FileSourceScan. This PR will still work if, for example, you are using BatchScan. ### Why are the changes needed? Some queries can benefit from having it's scalar subquery in the filter evaluated while planning so that the scalar result (from the subquery) can be push down. For example, a query like `select * from t2 where b > (select max(a) from t1) ` where t1 is a small table but t2 is a very large table can benefit if we first evaluate the subquery then push down the result to the pushed filter (instead of having the subquery in the post scan filter) ### Does this PR introduce _any_ user-facing change? Yes. A new conf is added (which is disabled by default) ### How was this patch tested? Unit tested and manually tested ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
