dongjoon-hyun commented on a change in pull request #26225: [SPARK-29568][SS]
Stop existing running streams when a new stream is launched
URL: https://github.com/apache/spark/pull/26225#discussion_r342878274
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1087,6 +1087,14 @@ object SQLConf {
.checkValue(v => Set(1, 2).contains(v), "Valid versions are 1 and 2")
.createWithDefault(2)
+ val STOP_RUNNING_DUPLICATE_STREAM =
buildConf("spark.sql.streaming.stopExistingDuplicateStream")
+ .doc("Running two streams using the same checkpoint location concurrently
is not supported. " +
+ "In the case where multiple streams are started on different
SparkSessions, access to the " +
+ "older stream's SparkSession may not be possible, and the stream may
have turned into a " +
+ "zombie stream. When this flag is true, we will stop the old stream to
start the new one.")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
+1 for the release notes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
