dongjoon-hyun edited a comment on issue #24339: [SPARK-27436][SQL] Add spark.sql.optimizer.nonExcludedRules URL: https://github.com/apache/spark/pull/24339#issuecomment-482071376 Thank you for review, @MaxGekk and @gatorsmile . @gatorsmile The built-in list is used to protect only Apache Spark optimizer rules. In addition, the problem is that we cannot change it at all without building from the sources (after changing the built-in list or updating a custom `Optimizer`) and distributing to the cluster. We know `set spark.sql.optimizer.excludedRules` is convenient for the dev environments for debugging and also a good safe-guard to remove malfunctioning optimizers in the production environments. However, in the same way, this is too dangerous for the production environments because all user can use this and make a damage to the whole executions mistakenly. We gave a too powerful sword, `set spark.sql.optimizer.excludedRules`, to the production users. Technically, we need to provide a shield, too. BTW, at the beginning of `spark.sql.optimizer.excludedRules`, I already suggested to make `spark.sql.optimizer.excludedRules` as a static configuration to the author of that PR, but it's not accepted because the dynamic debugging was preferred by the author of the that PR.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
