Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21701#discussion_r200771488
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
---
@@ -115,8 +130,20 @@ object OffsetSeqMetadata extends Logging {
case None =>
// For backward compatibility, if a config was not recorded in
the offset log,
- // then log it, and let the existing conf value in SparkSession
prevail.
- logWarning (s"Conf '$confKey' was not found in the offset log,
using existing value")
+ // then either inject a default value (if specified in
`relevantSQLConfDefaultValues`) or
+ // let the existing conf value in SparkSession prevail.
+ relevantSQLConfDefaultValues.get(confKey) match {
+
+ case Some(defaultValue) =>
+ sessionConf.set(confKey, defaultValue)
+ logWarning(s"Conf '$confKey' was not found in the offset
log, " +
+ s"using default value '$defaultValue'")
+
+ case None =>
+ logWarning(s"Conf '$confKey' was not found in the offset
log, " +
+ s"using existing value '${sessionConf.get(confKey, null)}")
--- End diff --
`'${sessionConf.get(confKey, null)}`? What's this supposed to be? It seems
weird to output `null` here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
