squito commented on issue #23677: [SPARK-26755] : Optimize Spark Scheduler to dequeue speculative tasks… URL: https://github.com/apache/spark/pull/23677#issuecomment-458647926 I haven't looked very closely yet, but general idea makes sense, and results are very compelling. High level question though -- any reason `TaskSetManager.canFetchMoreResults` needs to lock the `TaskSchedulerImpl`? Couldn't we protect `tsm.totalResultSize` and `tsm.calculatedTasks` with something local to the `tsm`, which would be entirely separate from the lock needed for speculation? I'm really just brainstorming at this point -- that change might actually be a lot harder to reason about.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
