Ngone51 commented on pull request #33872: URL: https://github.com/apache/spark/pull/33872#issuecomment-965452097
Actually, `taskIdToTaskSetManager` is still under the protection of `TaskSchedulerImpl`'s lock. `taskIdToTaskSetManager` has no problem here. The real problem is the thread-safety of `TaskSetManager.taskInfos`. Ideally, as for this issue, `handleSuccessfulTask` should return immediately when `taskInfo.finished=true` (caused by executorLost). That said, we should be careful to address thread-safety issue of `taskInfo` since it's referenced at the hot code path. I'd like to investigate more to see if we have other better solutions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
