Ngone51 commented on pull request #30795: URL: https://github.com/apache/spark/pull/30795#issuecomment-747858175
> so there were changes to attempt to help with this dealing with unschedulableTaskSets that went in a while ago. seems like they are trying to solve similar things so we should see overlap. Yeah, I also noticed the change(https://github.com/apache/spark/pull/28287). This PR can reduce the times of getting into the bad situation mentioned in https://github.com/apache/spark/pull/28287. As this PR essentially replaces those excluded executors with new healthy executors. Thus, a taskset can get more opportunities to launch tasks. Besides, we'd launch the new healthy executor early in this PR comparing to the solution in https://github.com/apache/spark/pull/28287, which helps improve the scheduling efficiency. However, this PR can not 100% replace https://github.com/apache/spark/pull/28287. Because we don't handle `SparkListenerExecutorExcludedForStage`/`SparkListenerNodeExcludedForStage` in this PR. So the taskset can still get completely excluded to launch tasks. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
