squito commented on issue #24497: [SPARK-27630][CORE]Stage retry causes totalRunningTasks calculation to be negative URL: https://github.com/apache/spark/pull/24497#issuecomment-489195300 > ExecutorAllocationListener stores the stage related information, the hash map key is the stage id, and there is no stage attempt id, the statistical information is for the current active stage. well ... that doesn't mean that's the right behavior. From the ExecutionAllocationManager's point of view, running tasks in a zombie taskset are still using up slots on executors, right? Eg. suppose you have five zombie tasks, and then another taskset with two more tasks comes along -- shouldn't the executor allocation manager try to request more executors for two more task slots? > In SPARK-11334, I saw a description added to stageIdToNumRunningTask >>Number of running tasks per stage including speculative tasks. >>Should be 0 when no stages are active. >If the stage's attempts are all zombie, their information should not be counted to the current active stage. I wouldn't assume that "active" is used consistently in the codebase with the way I defined it above. Unfortunately I think the only term used consistently is zombie vs. non-zombie. Indeed the part you quote is discussing stages, not stage attempts, so I really doubt it means the same thing.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
