squito commented on a change in pull request #23951: [SPARK-27038][CORE][YARN]
Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r264290976
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
##########
@@ -184,11 +184,23 @@ private[spark] class TaskSetManager(
t.epoch = epoch
}
+ // An array to store preferred location and its task index
+ private val locationWithTaskIndex: ArrayBuffer[(String, Int)] = new
ArrayBuffer[(String, Int)]()
+ private val addTaskStartTime = System.nanoTime()
// Add all our tasks to the pending lists. We do this in reverse order
// of task index so that tasks with low indices get launched first.
for (i <- (0 until numTasks).reverse) {
- addPendingTask(i)
+ addPendingTask(i, true)
}
+ // Convert preferred location list to rack list in one invocation and zip
with the origin index
+ private val rackWithTaskIndex =
sched.getRacksForHosts(locationWithTaskIndex.map(_._1).toList)
Review comment:
On de-duping -- I know you can't do it blindly, as then you won't correctly
keep track of which task goes with which host. But if you look at my proposed
change, I think it keeps track of the correct mapping back to the original task
indices (it certainly should be possible, anyway, even if that exact version
doesn't work).
Looking at `CachedDNSToSwitchMapping`, if you repeat the same host 1000
times, if that host is not in the cache yet, it will repeat that lookup 1000
times.
[`getUncachedHosts`](https://github.com/apache/hadoop/blob/ebb5fa115b842a3bcb747e881ca2a8fe4c716d07/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/CachedDNSToSwitchMapping.java#L116)
does a lookup to check if its in the cache, but if not it'll just pass each
copy on to the `uncachedHosts` variable. Then that just gets passed through to
[`rawMapping.resolve()`](https://github.com/apache/hadoop/blob/ebb5fa115b842a3bcb747e881ca2a8fe4c716d07/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/CachedDNSToSwitchMapping.java#L119).
(Perhaps `getUncachedHosts` really should return a `Set<String>`.)
so after looking at that code, I'm more convinced de-duping would help you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
