wineternity commented on code in PR #38702:
URL: https://github.com/apache/spark/pull/38702#discussion_r1044596644
##########
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:
##########
@@ -674,22 +674,30 @@ private[spark] class AppStatusListener(
delta
}.orNull
- val (completedDelta, failedDelta, killedDelta) = event.reason match {
+ // SPARK-41187: For `SparkListenerTaskEnd` with `Resubmitted` reason,
which is raised by
+ // executor lost, it can lead to negative `LiveStage.activeTasks` since
there's no
+ // corresponding `SparkListenerTaskStart` event for each of them. The
negative activeTasks
+ // will make the stage always remains in the live stage list as it can
never meet the
+ // condition activeTasks == 0. This in turn causes the dead executor to
never be retained
+ // if that live stage's submissionTime is less than the dead executor's
removeTime.
+ val (completedDelta, failedDelta, killedDelta, activeTasksDelta) =
event.reason match {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
