Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21063#discussion_r184241625
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala
---
@@ -111,7 +112,12 @@ trait ProgressReporter extends Logging {
logDebug("Starting Trigger Calculation")
lastTriggerStartTimestamp = currentTriggerStartTimestamp
currentTriggerStartTimestamp = triggerClock.getTimeMillis()
- currentStatus = currentStatus.copy(isTriggerActive = true)
+ // isTriggerActive field is kept false for ContinuousExecution
+ // since it is tied to MicroBatchExecution
+ this match {
--- End diff --
nit: someone may have a concern that a trait needs to be aware of actual
implementation.
There looks like two options:
1. extract method to only update currentStatus for starting trigger
defaulting to `isTriggerActive = true`, and let `ContinuousExecution` overrides
the method.
2. just override `startTrigger()` in `ContinuousExecution`, and call
`super.startTrigger()` and update currentStatus once again. It might open very
small window for other threads to read invalid status information
(isTriggerActive = true), but will require less change if it is acceptable.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
