[jira] [Updated] (YARN-4440) FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time
[
https://issues.apache.org/jira/browse/YARN-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated YARN-4440:
Attachment: YARN-4440.003.patch
Modify the error.
> FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time
> -
>
> Key: YARN-4440
> URL: https://issues.apache.org/jira/browse/YARN-4440
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: YARN-4440.001.patch, YARN-4440.002.patch,
> YARN-4440.003.patch
>
>
> It seems there is a bug on {{FSAppAttempt#getAllowedLocalityLevelByTime}}
> method
> {code}
> // default level is NODE_LOCAL
> if (! allowedLocalityLevel.containsKey(priority)) {
> allowedLocalityLevel.put(priority, NodeType.NODE_LOCAL);
> return NodeType.NODE_LOCAL;
> }
> {code}
> If you first invoke this method, it doesn't init time in
> lastScheduledContainer and this will lead to execute these code for next
> invokation:
> {code}
> // check waiting time
> long waitTime = currentTimeMs;
> if (lastScheduledContainer.containsKey(priority)) {
> waitTime -= lastScheduledContainer.get(priority);
> } else {
> waitTime -= getStartTime();
> }
> {code}
> the waitTime will subtract to FsApp startTime, and this will be easily more
> than the delay time and allowedLocality degrade. Because FsApp startTime will
> be start earlier than currentTimeMs. So we should add the initial time of
> priority to prevent comparing with FsApp startTime and allowedLocalityLevel
> degrade. And this problem will have more negative influence for small-jobs.
> The YARN-4399 also discuss some problem in aspect of locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4440) FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time
[
https://issues.apache.org/jira/browse/YARN-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated YARN-4440:
Attachment: YARN-4440.002.patch
Resolve the remained error.
> FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time
> -
>
> Key: YARN-4440
> URL: https://issues.apache.org/jira/browse/YARN-4440
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: YARN-4440.001.patch, YARN-4440.002.patch
>
>
> It seems there is a bug on {{FSAppAttempt#getAllowedLocalityLevelByTime}}
> method
> {code}
> // default level is NODE_LOCAL
> if (! allowedLocalityLevel.containsKey(priority)) {
> allowedLocalityLevel.put(priority, NodeType.NODE_LOCAL);
> return NodeType.NODE_LOCAL;
> }
> {code}
> If you first invoke this method, it doesn't init time in
> lastScheduledContainer and this will lead to execute these code for next
> invokation:
> {code}
> // check waiting time
> long waitTime = currentTimeMs;
> if (lastScheduledContainer.containsKey(priority)) {
> waitTime -= lastScheduledContainer.get(priority);
> } else {
> waitTime -= getStartTime();
> }
> {code}
> the waitTime will subtract to FsApp startTime, and this will be easily more
> than the delay time and allowedLocality degrade. Because FsApp startTime will
> be start earlier than currentTimeMs. So we should add the initial time of
> priority to prevent comparing with FsApp startTime and allowedLocalityLevel
> degrade. And this problem will have more negative influence for small-jobs.
> The YARN-4399 also discuss some problem in aspect of locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4440) FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time
[
https://issues.apache.org/jira/browse/YARN-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated YARN-4440:
Attachment: YARN-4440.001.patch
> FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time
> -
>
> Key: YARN-4440
> URL: https://issues.apache.org/jira/browse/YARN-4440
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: YARN-4440.001.patch
>
>
> It seems there is a bug on {{FSAppAttempt#getAllowedLocalityLevelByTime}}
> method
> {code}
> // default level is NODE_LOCAL
> if (! allowedLocalityLevel.containsKey(priority)) {
> allowedLocalityLevel.put(priority, NodeType.NODE_LOCAL);
> return NodeType.NODE_LOCAL;
> }
> {code}
> If you first invoke this method, it doesn't init time in
> lastScheduledContainer and this will lead to execute these code for next
> invokation:
> {code}
> // check waiting time
> long waitTime = currentTimeMs;
> if (lastScheduledContainer.containsKey(priority)) {
> waitTime -= lastScheduledContainer.get(priority);
> } else {
> waitTime -= getStartTime();
> }
> {code}
> the waitTime will subtract to FsApp startTime, and this will be easily more
> than the delay time and allowedLocality degrade. Because FsApp startTime will
> be start earlier than currentTimeMs. So we should add the initial time of
> priority to prevent comparing with FsApp startTime and allowedLocalityLevel
> degrade. And this problem will have more negative influence for small-jobs.
> The YARN-4399 also discuss some problem in aspect of locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
