dongjoon-hyun opened a new pull request #24339: [SPARK-27436][SQL] Add spark.sql.optimizer.nonExcludedRules URL: https://github.com/apache/spark/pull/24339 ## What changes were proposed in this pull request? Currently, users can remove optimizer rule dynamically. This PR aims to add a new feature with `spark.sql.optimizer.nonExcludedRules` static configuration to prevent accidental rule exclusion by users in SQL environment dynamically. This is useful for admin to protect custom optimizer rules. **AS IS** ```sql $ bin/spark-sql \ --conf spark.sql.optimizer.planChangeLog.level=ERROR \ --conf spark.sql.optimizer.planChangeLog.batches=OptimizeLimitZero \ --conf spark.sql.optimizer.planChangeLog.rules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero Spark master: local[*], Application Id: local-1554952359955 spark-sql> select 1 limit 0; 19/04/10 20:12:46 ERROR BaseSessionStateBuilder$$anon$2: === Applying Rule org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero === !GlobalLimit 0 LocalRelation <empty>, [1#0] !+- LocalLimit 0 ! +- Project [1 AS 1#0] ! +- OneRowRelation 19/04/10 20:12:46 ERROR BaseSessionStateBuilder$$anon$2: === Result of Batch OptimizeLimitZero === !GlobalLimit 0 LocalRelation <empty>, [1#0] !+- LocalLimit 0 ! +- Project [1 AS 1#0] ! +- OneRowRelation Time taken: 1.426 seconds spark-sql> set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero; 19/04/10 20:12:48 ERROR BaseSessionStateBuilder$$anon$2: Batch OptimizeLimitZero has no effect. spark.sql.optimizer.excludedRules org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero Time taken: 0.26 seconds, Fetched 1 row(s) spark-sql> select 1 limit 0; Time taken: 0.226 seconds ``` **AFTER (with spark.sql.optimizer.nonExcludedRules)** ```sql $ bin/spark-shell \ --conf spark.sql.optimizer.planChangeLog.level=ERROR \ --conf spark.sql.optimizer.planChangeLog.batches=OptimizeLimitZero \ --conf spark.sql.optimizer.planChangeLog.rules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero \ --conf spark.sql.optimizer.nonExcludedRules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero Spark master: local[*], Application Id: local-1554952378462 spark-sql> set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero; 19/04/10 20:13:04 ERROR BaseSessionStateBuilder$$anon$2: Batch OptimizeLimitZero has no effect. 19/04/10 20:13:04 ERROR BaseSessionStateBuilder$$anon$2: Batch OptimizeLimitZero has no effect. spark.sql.optimizer.excludedRules org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero Time taken: 1.563 seconds, Fetched 1 row(s) spark-sql> select 1 limit 0; 19/04/10 20:13:07 ERROR BaseSessionStateBuilder$$anon$2: === Applying Rule org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero === !GlobalLimit 0 LocalRelation <empty>, [1#4] !+- LocalLimit 0 ! +- Project [1 AS 1#4] ! +- OneRowRelation 19/04/10 20:13:07 ERROR BaseSessionStateBuilder$$anon$2: === Result of Batch OptimizeLimitZero === !GlobalLimit 0 LocalRelation <empty>, [1#4] !+- LocalLimit 0 ! +- Project [1 AS 1#4] ! +- OneRowRelation ``` ## How was this patch tested? Pass the Jenkins with a newly added test case.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
