tgravescs commented on a change in pull request #32526:
URL: https://github.com/apache/spark/pull/32526#discussion_r637971172
##########
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##########
@@ -920,7 +931,7 @@ private[spark] class ExecutorAllocationManager(
val attempts = resourceProfileIdToStageAttempt.getOrElse(rp,
Set.empty).toSeq
// attempts is a Set, change to Seq so we keep all values
attempts.map { attempt =>
- stageAttemptToNumRunningTask.getOrElseUpdate(attempt, 0)
+ stageAttemptToNumRunningTask.getOrElse(attempt, 0)
Review comment:
guess there is more in the jira:
> If spark-dynamic-executor-allocation thread calls schedule() after a
SparkListenerTaskEnd event for the last task in a stage
but before SparkListenerStageCompleted event for the stage, then
stageAttemptToNumRunningTask will not be cleaned up properly.
I'm not following this though because onTaskEnd both removes the stage from
stageAttemptToNumRunningTask as well as removes the stageAttempt from
resourceProfileIdToStageAttempt. so when this is called it shouldn't be in
resourceProfileIdToStageAttempt. Is there a case we are hitting that it is in
there, like multiple stages using that resource profile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
