Lin Yiqun created YARN-4440:

             Summary: FSAppAttempt#getAllowedLocalityLevelByTime should init 
the lastScheduler time
                 Key: YARN-4440
             Project: Hadoop YARN
          Issue Type: Bug
          Components: fairscheduler
    Affects Versions: 2.7.1
            Reporter: Lin Yiqun
            Assignee: Lin Yiqun

It seems there is a bug on {{FSAppAttempt#getAllowedLocalityLevelByTime}} method
// default level is NODE_LOCAL
    if (! allowedLocalityLevel.containsKey(priority)) {
      allowedLocalityLevel.put(priority, NodeType.NODE_LOCAL);
      return NodeType.NODE_LOCAL;
If you first invoke this method, it doesn't init  time in 
lastScheduledContainer and this will lead to execute these code for next 
    // check waiting time
    long waitTime = currentTimeMs;
    if (lastScheduledContainer.containsKey(priority)) {
      waitTime -= lastScheduledContainer.get(priority);
    } else {
      waitTime -= getStartTime();
the waitTime will subtract to FsApp startTime, and this will be easily more 
than the delay time and allowedLocality degrade. Because FsApp startTime will 
be start earlier than currentTimeMs. So we should add the initial time of 
priority to prevent comparing with FsApp startTime and allowedLocalityLevel 
degrade. And this problem will have more negative influence for small-jobs. The 
YARN-4399 also discuss some problem in aspect of locality.

This message was sent by Atlassian JIRA

Reply via email to