[
https://issues.apache.org/jira/browse/YARN-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780125#comment-16780125
]
Lantao Jin edited comment on YARN-9332 at 2/28/19 5:39 AM:
-----------------------------------------------------------
[~cheersyang] Yes, Spark has to invoke this method in a deep loop.
{code}
for (i <- (0 until numTasks).reverse) {
addPendingTask(i, true)
}
...
private[spark] def addPendingTask(index: Int) {
for (loc <- tasks(index).preferredLocations) {
...
for (rack <- sched.getRackForHost(loc.host)) { //<---- invoke here
...
}
}
...
}
{code}
I am preparing a ticket for Spark, it could save 15~20 seconds when launching
big task set.
was (Author: cltlfcjin):
[~cheersyang] Yes, Spark has to invoke this method in a deep loop.
{code}
for (i <- (0 until numTasks).reverse) {
addPendingTask(i, true)
}
...
private[spark] def addPendingTask(index: Int) {
for (loc <- tasks(index).preferredLocations) {
...
for (rack <- sched.getRackForHost(loc.host)) { //<---- invoke here
...
}
}
...
}
{code}
I am preparing a ticket for Spark, it could save 15~20 seconds when launching
big task.
> RackResolver tool should accept multiple hosts
> ----------------------------------------------
>
> Key: YARN-9332
> URL: https://issues.apache.org/jira/browse/YARN-9332
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 2.9.2, 3.0.3, 2.8.5, 2.7.7, 3.1.2
> Reporter: Lantao Jin
> Assignee: Lantao Jin
> Priority: Minor
> Attachments: YARN-9332.001.patch
>
>
> RackResolver as a public rack resolver tool only offers a method {{public
> static Node resolve(String hostName)}} which only accepts one host a time.
> Actually the internal implementation class {{DNSToSwitchMapping}} always
> accept a host list as its input and return a list of resolved racks. That's
> cause the invoker like Spark takes a long time to resolve the rack info when
> handling abundant tasks (a mass of loops to execute script to resolve rack
> info).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
