pgandhi999 commented on issue #23677: [SPARK-26755][SCHEDULER] : Optimize Spark Scheduler to dequeue speculative tasks… URL: https://github.com/apache/spark/pull/23677#issuecomment-458722096 @squito That is a good point to look into. The lock at the task scheduler level was introduced to `TaskSetManager.canFetchMoreResults` in https://github.com/apache/spark/commit/c306555f491e45ef870f58938af397f9ec5f166a with the corresponding JIRA https://issues.apache.org/jira/browse/SPARK-5219. The JIRA is not descriptive enough so I do not know the exact reason as to why was the lock added.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
