Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/19534
@jiangxb1987 - yes that is the issue and you are right, we can avoid it by
checking if the stageId is valid when we get a task end event. But I like this
approach better because we can clean up the hack which sets the
`numRunningTasks` to 0 when stage ends and also it is inline with the way we
are doing bookkeeping in ExecutorAllocationListener i.e, keeping entry per
stage. Let me know what you think.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
