Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22209#discussion_r214410470
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -350,11 +350,20 @@ private[spark] class AppStatusListener(
val e = it.next()
if (job.stageIds.contains(e.getKey()._1)) {
val stage = e.getValue()
- stage.status = v1.StageStatus.SKIPPED
- job.skippedStages += stage.info.stageId
- job.skippedTasks += stage.info.numTasks
- it.remove()
- update(stage, now)
+ if (v1.StageStatus.PENDING.equals(stage.status)) {
--- End diff --
> In the previous behaviour, it would have marked the stages that were
ACTIVE as SKIPPED
That sounds right, but that was also incorrect behavior. I think we need to
take a look at all the "what happens when events are dropped" cases, and that's
better done separately. It's much less likely for events to be dropped here
nowadays...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
