Ming Ma created YARN-4690:
-----------------------------
Summary: Skip object allocation in FSAppAttempt#getResourceUsage
when possible
Key: YARN-4690
URL: https://issues.apache.org/jira/browse/YARN-4690
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Ming Ma
YARN-2768 addresses an important bottleneck. Here is another similar instance
where object allocation in Resources#subtract will slow down the fair
scheduler's event processing thread.
{noformat}
org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java)
org.apache.hadoop.yarn.util.Records.newRecord(Records.java)
org.apache.hadoop.yarn.util.resource.Resources.createResource(Resources.java)
org.apache.hadoop.yarn.util.resource.Resources.clone(Resources.java)
org.apache.hadoop.yarn.util.resource.Resources.subtract(Resources.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.getResourceUsage(FSAppAttempt.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.getResourceUsage(FSLeafQueue.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FairSharePolicy$FairShareComparator.compare(FairSharePolicy.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FairSharePolicy$FairShareComparator.compare(FairSharePolicy.java)
java.util.TimSort.binarySort(TimSort.java)
java.util.TimSort.sort(TimSort.java)
java.util.TimSort.sort(TimSort.java)
java.util.Arrays.sort(Arrays.java)
java.util.Collections.sort(Collections.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java)
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java)
org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.handle(ResourceSchedulerWrapper.java)
org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.handle(ResourceSchedulerWrapper.java)
{noformat}
One way to fix it is to return {{getCurrentConsumption()}} if there is no
preemption which is the normal case. This means {{getResourceUsage}} method
will return reference to {{FSAppAttempt}}'s internal resource object. But that
should be ok as {{getResourceUsage}} doesn't expect the caller to modify the
object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
