venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r688678951
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
}
}
+ private def killedTaskSummaryForSpeculationStageSummary(
+ reason: TaskEndReason,
+ oldSummary: Map[String, Int],
+ isSpeculative: Boolean): Map[String, Int] = {
+ reason match {
+ case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary.updated("speculated attempt succeeded",
+ oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+ }
+ // If the stage is finished and speculative tasks get killed, then the
+ // kill reason is "stage finished"
+ case k: TaskKilled if k.reason.contains("Stage finished") =>
+ if (isSpeculative) {
+ oldSummary.updated("original attempt succeeded",
+ oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+ } else {
+ oldSummary
Review comment:
@sarutak Do you mean handling the case where the task that got killed
has a speculative attempt but got killed because of the speculative attempt
succeeded? I think this probably would be difficult to address as the stage is
finished and all the in memory data structures would be cleaned up, only the
`AppStatusStore` would have the information.
I don't think it is worth the effort to handle that checking
`AppStatusStore`. In that case, the `speculation attempt succeeded` count can
be one less, it is not a huge discrepancy IMO.
But in the case of zombie task attempts the counters can get completely
wrong. May be we can reset the counters to zero in the case of failed/killed
task attempts. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
