MaxGekk opened a new pull request #27092: [SPARK-30416][SQL] Log a warning for deprecated SQL config in `set()` and `unset()` URL: https://github.com/apache/spark/pull/27092 ### What changes were proposed in this pull request? 1. Put all deprecated SQL configs the map `SQLConf.deprecatedSQLConfigs` with extra info about when configs were deprecated and additional comments that explain why a config was deprecated, what an user can use instead of it. Here is the list of already deprecated configs: - `spark.sql.hive.verifyPartitionPath` - `spark.sql.execution.pandas.respectSessionTimeZone` - `spark.sql.legacy.execution.pandas.groupedMap.assignColumnsByName` 2. Output warning in `set()` and `unset()` about deprecated SQL configs ### Why are the changes needed? This should improve UX with Spark SQL and notify users about already deprecated SQL configs. ### Does this PR introduce any user-facing change? Yes, before: ``` spark-sql> set spark.sql.hive.verifyPartitionPath=true; spark.sql.hive.verifyPartitionPath true ``` After: ``` spark-sql> set spark.sql.hive.verifyPartitionPath=true; 20/01/03 21:28:17 WARN RuntimeConfig: The SQL config 'spark.sql.hive.verifyPartitionPath' has been deprecated in Spark v3.0.0 and may be removed in the future. This config will be replaced by spark.files.ignoreMissingFiles. spark.sql.hive.verifyPartitionPath true ``` ### How was this patch tested? Add new test which registers new log appender and catches all logging to check that `set()` and `unset()` log any warning.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
