xumingming commented on PR #57901: URL: https://github.com/apache/spark/pull/57901#issuecomment-5252880369
@cloud-fan Can you take a look at this one? The change here is not so elegant, it tries to undo the pushdown by other rules, but the benefit it gives us is considerable, it can turn full table scan into a fast one partition scan. Before this change the rule choose to rebuild the CTE child from a old snapshot with new predicate generated from the predicates collected from CTE reference site, it avoids stacking repeated predicates on top of CTE body, but it silently discards all the useful predicates other rule infers. In this change, we do not avoid stacking predicates by using snapshotted CTE body, instead we avoid by removing the already pushed predicates away before pushing new predicates, it is not so elegant, but the code involved is not too much, maintainable in my opinion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
