[
https://issues.apache.org/jira/browse/YARN-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993156#comment-13993156
]
Chris Riccomini commented on YARN-2027:
---------------------------------------
So, running this request with memMb=3584, cpuCores=1, containers=32:
{code}
protected def requestContainers(memMb: Int, cpuCores: Int, containers: Int) {
info("Requesting %d container(s) with %dmb of memory" format (containers,
memMb))
val capability = Records.newRecord(classOf[Resource])
val priority = Records.newRecord(classOf[Priority])
priority.setPriority(0)
capability.setMemory(memMb)
capability.setVirtualCores(cpuCores)
def getHosts = {
val hosts = getNextRoundRobinHosts
System.err.println(hosts.toList)
hosts
}
(0 until containers).foreach(idx => amClient.addContainerRequest(new
ContainerRequest(capability, getHosts, List("/default-rack").toArray[String],
priority, false)))
}
{code}
Prints this in the AM logs:
{noformat}
List(eat1-app857, eat1-app873, eat1-app880)
List(eat1-app854, eat1-app864, eat1-app872)
List(eat1-app852, eat1-app873, eat1-app880)
List(eat1-app854, eat1-app880, eat1-app867)
List(eat1-app875, eat1-app852, eat1-app873)
List(eat1-app875, eat1-app852, eat1-app872)
List(eat1-app873, eat1-app859, eat1-app880)
List(eat1-app854, eat1-app873, eat1-app864)
List(eat1-app852, eat1-app874, eat1-app875)
List(eat1-app864, eat1-app859, eat1-app880)
List(eat1-app874, eat1-app872, eat1-app875)
List(eat1-app874, eat1-app873, eat1-app864)
List(eat1-app873, eat1-app859, eat1-app858)
List(eat1-app874, eat1-app873, eat1-app854)
List(eat1-app867, eat1-app880, eat1-app872)
List(eat1-app859, eat1-app875, eat1-app880)
List(eat1-app875, eat1-app872, eat1-app864)
List(eat1-app875, eat1-app867, eat1-app852)
List(eat1-app857, eat1-app852, eat1-app867)
List(eat1-app872, eat1-app854, eat1-app858)
List(eat1-app852, eat1-app872, eat1-app858)
List(eat1-app880, eat1-app873, eat1-app857)
List(eat1-app859, eat1-app871, eat1-app874)
List(eat1-app880, eat1-app874, eat1-app865)
List(eat1-app867, eat1-app873, eat1-app875)
List(eat1-app857, eat1-app858, eat1-app852)
List(eat1-app857, eat1-app867, eat1-app873)
List(eat1-app857, eat1-app871, eat1-app854)
List(eat1-app874, eat1-app865, eat1-app873)
List(eat1-app852, eat1-app880, eat1-app858)
List(eat1-app875, eat1-app873, eat1-app871)
List(eat1-app854, eat1-app880, eat1-app865)
{noformat}
With DEBUG logging in the RM logs (with no other job on the grid), I see:
{noformat}
21:18:02,958 DEBUG AppSchedulingInfo:135 - update:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 32, Location: *, Relax Locality: false}
21:18:02,958 DEBUG ActiveUsersManager:68 - User my-job-name added to
activeUsers, currently: 1
21:18:02,959 DEBUG CapacityScheduler:704 - allocate: post-update
21:18:02,959 DEBUG SchedulerApplicationAttempt:328 - showRequests:
application=application_1399581102453_0003 headRoom=<memory:736256, vCores:0>
currentConsumption=1024
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 9, Location: eat1-app875, Relax
Locality: true}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 6, Location: eat1-app857, Relax
Locality: true}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 11, Location: eat1-app880, Relax
Locality: true}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 7, Location: eat1-app854, Relax
Locality: true}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 32, Location: /default-rack, Relax
Locality: true}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app858, Relax
Locality: true}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 32, Location: *, Relax Locality: false}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 7, Location: eat1-app874, Relax
Locality: true}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 7, Location: eat1-app872, Relax
Locality: true}
21:18:02,959 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app859, Relax
Locality: true}
21:18:02,960 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 13, Location: eat1-app873, Relax
Locality: true}
21:18:02,960 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app864, Relax
Locality: true}
21:18:02,960 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 3, Location: eat1-app865, Relax
Locality: true}
21:18:02,960 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 9, Location: eat1-app852, Relax
Locality: true}
21:18:02,960 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 6, Location: eat1-app867, Relax
Locality: true}
21:18:02,960 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 3, Location: eat1-app871, Relax
Locality: true}
21:18:02,960 DEBUG CapacityScheduler:709 - allocate:
applicationAttemptId=appattempt_1399581102453_0003_000001 #ask=16
21:18:03,009 DEBUG AsyncDispatcher:164 - Dispatching the event
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeStatusEvent.EventType:
STATUS_UPDATE
21:18:03,009 DEBUG RMNodeImpl:373 - Processing eat1-app863:35408 of type
STATUS_UPDATE
21:18:03,009 DEBUG AsyncDispatcher:164 - Dispatching the event
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeUpdateSchedulerEvent.EventType:
NODE_UPDATE
21:18:03,009 DEBUG CapacityScheduler:754 - nodeUpdate: eat1-app863:35408
clusterResources: <memory:737280, vCores:360>
21:18:03,009 DEBUG CapacityScheduler:785 - Node being looked for scheduling
eat1-app863:35408 availableResource: <memory:49152, vCores:24>
21:18:03,009 DEBUG CapacityScheduler:828 - Trying to schedule on node:
eat1-app863, available: <memory:49152, vCores:24>
21:18:03,009 DEBUG ParentQueue:559 - Trying to assign containers to child-queue
of root
21:18:03,009 DEBUG ParentQueue:690 - printChildQueues - queue: root
child-queues: root.default(0.0013888889),
21:18:03,009 DEBUG ParentQueue:652 - Trying to assign to queue: root.default
stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024,
vCores:1>usedCapacity=0.0013888889, absoluteUsedCapacity=0.0013888889,
numApps=1, numContainers=1
21:18:03,009 DEBUG LeafQueue:807 - assignContainers: node=eat1-app863
#applications=1
21:18:03,009 DEBUG LeafQueue:826 - pre-assignContainers for application
application_1399581102453_0003
21:18:03,009 DEBUG SchedulerApplicationAttempt:328 - showRequests:
application=application_1399581102453_0003 headRoom=<memory:736256, vCores:0>
currentConsumption=1024
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 9, Location: eat1-app875, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 6, Location: eat1-app857, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 11, Location: eat1-app880, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 7, Location: eat1-app854, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 32, Location: /default-rack, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app858, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 32, Location: *, Relax Locality: false}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 7, Location: eat1-app874, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 7, Location: eat1-app872, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app859, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 13, Location: eat1-app873, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app864, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 3, Location: eat1-app865, Relax
Locality: true}
21:18:03,010 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 9, Location: eat1-app852, Relax
Locality: true}
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 6, Location: eat1-app867, Relax
Locality: true}
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 3, Location: eat1-app871, Relax
Locality: true}
21:18:03,011 DEBUG LeafQueue:1062 - User limit computation for my-job-name in
queue default userLimit=100 userLimitFactor=1.0 required: <memory:3584,
vCores:1> consumed: <memory:1024, vCores:1> limit: <memory:737280, vCores:1>
queueCapacity: <memory:737280, vCores:1> qconsumed: <memory:1024, vCores:1>
currentCapacity: <memory:737280, vCores:1> activeUsers: 1 clusterCapacity:
<memory:737280, vCores:360>
21:18:03,011 DEBUG LeafQueue:995 - Headroom calculation for user my-job-name:
userLimit=<memory:737280, vCores:1> queueMaxCap=<memory:737280, vCores:1>
consumed=<memory:1024, vCores:1> headroom=<memory:736256, vCores:0>
21:18:03,011 DEBUG LeafQueue:914 - post-assignContainers for application
application_1399581102453_0003
21:18:03,011 DEBUG SchedulerApplicationAttempt:328 - showRequests:
application=application_1399581102453_0003 headRoom=<memory:736256, vCores:0>
currentConsumption=1024
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 9, Location: eat1-app875, Relax
Locality: true}
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 6, Location: eat1-app857, Relax
Locality: true}
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 11, Location: eat1-app880, Relax
Locality: true}
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 7, Location: eat1-app854, Relax
Locality: true}
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 32, Location: /default-rack, Relax
Locality: true}
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app858, Relax
Locality: true}
21:18:03,011 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 32, Location: *, Relax Locality: false}
21:18:03,012 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 7, Location: eat1-app872, Relax
Locality: true}
21:18:03,012 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app859, Relax
Locality: true}
21:18:03,012 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 13, Location: eat1-app873, Relax
Locality: true}
21:18:03,012 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 5, Location: eat1-app864, Relax
Locality: true}
21:18:03,012 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 3, Location: eat1-app865, Relax
Locality: true}
21:18:03,012 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 9, Location: eat1-app852, Relax
Locality: true}
21:18:03,012 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 6, Location: eat1-app867, Relax
Locality: true}
21:18:03,012 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 3, Location: eat1-app871, Relax
Locality: true}
21:18:03,012 DEBUG ParentQueue:657 - Assigned to queue: root.default stats:
default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024,
vCores:1>usedCapacity=0.0013888889, absoluteUsedCapacity=0.0013888889,
numApps=1, numContainers=1 --> <memory:0, vCores:0>, NODE_LOCAL
{noformat}
The logs go on for quite a while like this, but this is the general gist of it.
This is very confusing, since I made 32 single requests all with different
random hosts, a hard-coded /default-rack, and relaxLocality set to false. I'm
seeing all kinds of crazy requests in the logs.
I'm assuming the reason for the expansion in the number of requests is because
of the AMRMClientImpl's expansion in addContainerRequest:
{code}
if (req.getNodes() != null) {
HashSet<String> dedupedNodes = new HashSet<String>(req.getNodes());
if(dedupedNodes.size() != req.getNodes().size()) {
Joiner joiner = Joiner.on(',');
LOG.warn("ContainerRequest has duplicate nodes: "
+ joiner.join(req.getNodes()));
}
for (String node : dedupedNodes) {
addResourceRequest(req.getPriority(), node, req.getCapability(), req,
true);
}
}
for (String rack : dedupedRacks) {
addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
true);
}
// Ensure node requests are accompanied by requests for
// corresponding rack
for (String rack : inferredRacks) {
addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
req.getRelaxLocality());
}
// Off-switch
addResourceRequest(req.getPriority(), ResourceRequest.ANY,
req.getCapability(), req, req.getRelaxLocality());
{code}
Still, the first request that is logged from the RM is this one:
{noformat}
21:18:03,058 DEBUG SchedulerApplicationAttempt:332 - showRequests:
application=application_1399581102453_0003 request={Priority: 0, Capability:
<memory:3584, vCores:1>, # Containers: 9, Location: eat1-app875, Relax
Locality: true}
{noformat}
Sure enough, I see 9 containers running on eat1-app875. Why is the RM behaving
in this way? I made 32 SINGLE requests, each with DIFFERENT hosts, with
relaxLocality OFF, but it seems that either the client or the scheduler is
bunching them together without my consent.
> YARN ignores host-specific resource requests
> --------------------------------------------
>
> Key: YARN-2027
> URL: https://issues.apache.org/jira/browse/YARN-2027
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
> Affects Versions: 2.4.0
> Environment: RHEL 6.1
> YARN 2.4
> Reporter: Chris Riccomini
>
> YARN appears to be ignoring host-level ContainerRequests.
> I am creating a container request with code that pretty closely mirrors the
> DistributedShell code:
> {code}
> protected def requestContainers(memMb: Int, cpuCores: Int, containers: Int)
> {
> info("Requesting %d container(s) with %dmb of memory" format (containers,
> memMb))
> val capability = Records.newRecord(classOf[Resource])
> val priority = Records.newRecord(classOf[Priority])
> priority.setPriority(0)
> capability.setMemory(memMb)
> capability.setVirtualCores(cpuCores)
> // Specifying a host in the String[] host parameter here seems to do
> nothing. Setting relaxLocality to false also doesn't help.
> (0 until containers).foreach(idx => amClient.addContainerRequest(new
> ContainerRequest(capability, null, null, priority)))
> }
> {code}
> When I run this code with a specific host in the ContainerRequest, YARN does
> not honor the request. Instead, it puts the container on an arbitrary host.
> This appears to be true for both the FifoScheduler and the CapacityScheduler.
> Currently, we are running the CapacityScheduler with the following settings:
> {noformat}
> <configuration>
> <property>
> <name>yarn.scheduler.capacity.maximum-applications</name>
> <value>10000</value>
> <description>
> Maximum number of applications that can be pending and running.
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
> <value>0.1</value>
> <description>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.resource-calculator</name>
>
> <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
> <description>
> The ResourceCalculator implementation to be used to compare
> Resources in the scheduler.
> The default i.e. DefaultResourceCalculator only uses Memory while
> DominantResourceCalculator uses dominant-resource to compare
> multi-dimensional resources such as Memory, CPU etc.
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.root.queues</name>
> <value>default</value>
> <description>
> The queues at the this level (root is the root queue).
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.root.default.capacity</name>
> <value>100</value>
> <description>Samza queue target capacity.</description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
> <value>1</value>
> <description>
> Default queue user limit a percentage from 0.0 to 1.0.
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
> <value>100</value>
> <description>
> The maximum capacity of the default queue.
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.root.default.state</name>
> <value>RUNNING</value>
> <description>
> The state of the default queue. State can be one of RUNNING or STOPPED.
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
> <value>*</value>
> <description>
> The ACL of who can submit jobs to the default queue.
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
> <value>*</value>
> <description>
> The ACL of who can administer jobs on the default queue.
> </description>
> </property>
> <property>
> <name>yarn.scheduler.capacity.node-locality-delay</name>
> <value>40</value>
> <description>
> Number of missed scheduling opportunities after which the
> CapacityScheduler
> attempts to schedule rack-local containers.
> Typically this should be set to number of nodes in the cluster, By
> default is setting
> approximately number of nodes in one rack which is 40.
> </description>
> </property>
> </configuration>
> {noformat}
> Digging into the code a bit (props to [~jghoman] for finding this), we have a
> theory as to why this is happening. It looks like
> RMContainerRequestor.addContainerReq adds three resource requests per
> container request: data-local, rack-local, and any:
> {code}
> protected void addContainerReq(ContainerRequest req) {
> // Create resource requests
> for (String host : req.hosts) {
> // Data-local
> if (!isNodeBlacklisted(host)) {
> addResourceRequest(req.priority, host, req.capability);
> }
> }
> // Nothing Rack-local for now
> for (String rack : req.racks) {
> addResourceRequest(req.priority, rack, req.capability);
> }
> // Off-switch
> addResourceRequest(req.priority, ResourceRequest.ANY, req.capability);
> }
> {code}
> The addResourceRequest method, in turn, calls addResourceRequestToAsk, which
> in turn calls ask.add(remoteRequest):
> {code}
> private void addResourceRequestToAsk(ResourceRequest remoteRequest) {
> // because objects inside the resource map can be deleted ask can end up
> // containing an object that matches new resource object but with different
> // numContainers. So exisintg values must be replaced explicitly
> if(ask.contains(remoteRequest)) {
> ask.remove(remoteRequest);
> }
> ask.add(remoteRequest);
> }
> {code}
> The problem is that the "ask" variable is a TreeSet:
> {code}
> private final Set<ResourceRequest> ask = new TreeSet<ResourceRequest>(
> new
> org.apache.hadoop.yarn.api.records.ResourceRequest.ResourceRequestComparator());
> {code}
> The ResourceRequestComparator sorts the TreeSet according to:
> {code}
> public int compare(ResourceRequest r1, ResourceRequest r2) {
> // Compare priority, host and capability
> int ret = r1.getPriority().compareTo(r2.getPriority());
> if (ret == 0) {
> String h1 = r1.getResourceName();
> String h2 = r2.getResourceName();
> ret = h1.compareTo(h2);
> }
> if (ret == 0) {
> ret = r1.getCapability().compareTo(r2.getCapability());
> }
> return ret;
> }
> {code}
> The first thing to note is that our resource requests all have the same
> priority, so the TreeSet is really sorted by resource name (host/rack). The
> resource names that are added as part of addContainerReq are host, rack, and
> any, which is denoted as "\*" (see above). The problem with this is that the
> TreeSet is going to sort the resource requests with the "\*" request first,
> even if the host request was added first in addContainerReq.
> {code}
> > import java.util.TreeSet
> > val set = new TreeSet[String]
> set: java.util.TreeSet[String] = []
> > set.add("eat1-app")
> > set
> res3: java.util.TreeSet[String] = [eat1-app]
> > set.add("*")
> > set
> res5: java.util.TreeSet[String] = [*, eat1-app]
> {code}
> From here on out, it seems to me that anything interacting with the "ask"
> TreeSet (including the allocation requests) will be using the most general
> resource request, not the most specific.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
