pgandhi999 commented on a change in pull request #23677: [SPARK-26755][SCHEDULER] : Optimize Spark Scheduler to dequeue speculative tasks… URL: https://github.com/apache/spark/pull/23677#discussion_r299164450
########## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ########## @@ -158,9 +158,21 @@ private[spark] class TaskSetManager( // Set containing all pending tasks (also used as a stack, as above). private val allPendingTasks = new ArrayBuffer[Int] - // Tasks that can be speculated. Since these will be a small fraction of total - // tasks, we'll just hold them in a HashSet. - private[scheduler] val speculatableTasks = new HashSet[Int] + // Set of pending tasks that can be speculated for each executor. + private[scheduler] var pendingSpeculatableTasksForExecutor = + new HashMap[String, HashSet[Int]] + + // Set of pending tasks that can be speculated for each host. + private[scheduler] var pendingSpeculatableTasksForHost = new HashMap[String, HashSet[Int]] + + // Set of pending tasks that can be speculated with no locality preferences. + private[scheduler] val pendingSpeculatableTasksWithNoPrefs = new HashSet[Int] + + // Set of pending tasks that can be speculated for each rack. + private[scheduler] var pendingSpeculatableTasksForRack = new HashMap[String, HashSet[Int]] + + // Set of all pending tasks that can be speculated. + private[scheduler] val allPendingSpeculatableTasks = new HashSet[Int] Review comment: makes sense @squito, I have refactored the code. Please let me know if it looks different from what you had thought earlier. Thank you. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
