maropu commented on a change in pull request #27057:
[SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed
SQL configs
URL: https://github.com/apache/spark/pull/27057#discussion_r362668497
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -170,6 +170,36 @@ object SQLConf {
}
}
+ /**
+ * Holds information about keys that have been removed.
+ *
+ * @param key The removed config key.
+ * @param version Version of Spark where key was removed.
+ * @param defaultValue The default config value. It can be used to notice
+ * users that they set non-default value to an already
removed config.
+ * @param comment Additional info regarding to the removed config.
+ */
+ case class RemovedConfig(key: String, version: String, defaultValue: String,
comment: String)
+
+ /**
+ * The map contains info about removed SQL configs. Keys are SQL config
names,
+ * map values contain extra information like the version in which the config
was removed,
+ * config's default value and a comment.
+ */
+ val removedSQLConfigs: Map[String, RemovedConfig] = {
+ val configs = Seq(
+ RemovedConfig("spark.sql.fromJsonForceNullableSchema", "3.0.0", "true",
+ "It was removed to prevent errors like SPARK-23173 for non-default
value."),
+ RemovedConfig(
+ "spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation",
"3.0.0", "false",
+ "It was removed to prevent loosing of users data for non-default
value."),
+ RemovedConfig("spark.sql.legacy.compareDateTimestampInTimestamp",
"3.0.0", "true",
+ "It was removed to prevent errors like SPARK-23549 for non-default
value.")
+ )
Review comment:
Looks a nice feature. Just a question; any policy about when these entries
will be removed in this list? We need to keep them forever, or we can remove
them sometime?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
