Lin Yiqun created YARN-4440:
-------------------------------
Summary: FSAppAttempt#getAllowedLocalityLevelByTime should init
the lastScheduler time
Key: YARN-4440
URL: https://issues.apache.org/jira/browse/YARN-4440
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.7.1
Reporter: Lin Yiqun
Assignee: Lin Yiqun
It seems there is a bug on {{FSAppAttempt#getAllowedLocalityLevelByTime}} method
{code}
// default level is NODE_LOCAL
if (! allowedLocalityLevel.containsKey(priority)) {
allowedLocalityLevel.put(priority, NodeType.NODE_LOCAL);
return NodeType.NODE_LOCAL;
}
{code}
If you first invoke this method, it doesn't init time in
lastScheduledContainer and this will lead to execute these code for next
invokation:
{code}
// check waiting time
long waitTime = currentTimeMs;
if (lastScheduledContainer.containsKey(priority)) {
waitTime -= lastScheduledContainer.get(priority);
} else {
waitTime -= getStartTime();
}
{code}
the waitTime will subtract to FsApp startTime, and this will be easily more
than the delay time and allowedLocality degrade. Because FsApp startTime will
be start earlier than currentTimeMs. So we should add the initial time of
priority to prevent comparing with FsApp startTime and allowedLocalityLevel
degrade. And this problem will have more negative influence for small-jobs. The
YARN-4399 also discuss some problem in aspect of locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
