weixiuli opened a new pull request, #36162: URL: https://github.com/apache/spark/pull/36162
### What changes were proposed in this pull request? Currently, no matter the efficiency of some tasks, as long as certain conditions are met, they will speculate, which is a huge waste of cluster resources. In production, some speculative tasks may come from some efficient tasks, and since efficient tasks are completed first, speculative tasks will eventually be killed, so there is no need to speculate on them. Unnecessary speculative tasks not only waste cluster resources but also interfere with the scheduling of other tasks. Therefore, we should first assess the inefficiency of the task by measuring the success of the task, and then decide whether to speculate. The low efficiency task should be considered, and the high efficiency task should not be considered, which is beneficial to the optimization of cluster resources. ### Why are the changes needed? Improve the speculation. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Add unittests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
