Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20622#discussion_r170392408
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala
---
@@ -266,6 +264,12 @@ class ContinuousExecution(
SQLExecution.withNewExecutionId(
sparkSessionForQuery, lastExecution)(lastExecution.toRdd)
}
+ } catch {
+ case t: Throwable
+ if StreamExecution.isInterruptionException(t) && state.get() ==
RECONFIGURING =>
+ stopSources()
+ sparkSession.sparkContext.cancelJobGroup(runId.toString)
--- End diff --
So we only swallow the exception when we are reconfiguration (btw always
add logging when swallowing exceptions to leave a trail for debugging), and
`stopSources()` and `cancelJobGroups()` can be finally as we want that as
invariant no matter what happens in this `runContinuous` method.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
