yeshengm opened a new pull request #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor URL: https://github.com/apache/spark/pull/25249 ## What changes were proposed in this pull request? Since in adaptive query processing (AQE), query plans are optimized on the fly during execution, a few `Once` rules can be problematic for such optimization since they can either generate wrong plan/unnecessary intermediate plan nodes. This PR enforces idempotence for "Once" batches that are supposed to run once. This is a key enabler for AQE re-optimization and can improve robustness for existing optimizer rules. Once batches that are currently not idempotent are marked in a blacklist. We will submit followup PRs to fix idempotence of these rules. ## How was this patch tested? Existing UTs. Failing Once rules are temporarily blacklisted.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
