[jira] [Updated] (YARN-3319) Implement a Fair SchedulerOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3319: -- Attachment: YARN-3319.39.patch > Implement a Fair SchedulerOrderingPolicy > > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch > > > Implement a Fair Comparator for the Scheduler Comparator Ordering Policy > which prefers to allocate to SchedulerProcesses with least current usage, > very similar to the FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > An implementation of a Scheduler Comparator for use with the Scheduler > Comparator Ordering Policy will be built with the below comparison for > ordering applications for container assignment (ascending) and for preemption > (descending) > Current resource usage - less usage is lesser > Submission time - earlier is lesser > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > name, which is lexically FIFO for that comparison (first submitted is lesser) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3319) Implement a Fair SchedulerOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3319: -- Attachment: YARN-3319.35.patch Apply after applying YARN-3318.35.patch > Implement a Fair SchedulerOrderingPolicy > > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch > > > Implement a Fair Comparator for the Scheduler Comparator Ordering Policy > which prefers to allocate to SchedulerProcesses with least current usage, > very similar to the FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > An implementation of a Scheduler Comparator for use with the Scheduler > Comparator Ordering Policy will be built with the below comparison for > ordering applications for container assignment (ascending) and for preemption > (descending) > Current resource usage - less usage is lesser > Submission time - earlier is lesser > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > name, which is lexically FIFO for that comparison (first submitted is lesser) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3319) Implement a Fair SchedulerOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3319: -- Description: Implement a Fair Comparator for the Scheduler Comparator Ordering Policy which prefers to allocate to SchedulerProcesses with least current usage, very similar to the FairScheduler's FairSharePolicy. The Policy will offer allocations to applications in a queue in order of least resources used, and preempt applications in reverse order (from most resources used). This will include conditional support for sizeBasedWeight style adjustment An implementation of a Scheduler Comparator for use with the Scheduler Comparator Ordering Policy will be built with the below comparison for ordering applications for container assignment (ascending) and for preemption (descending) Current resource usage - less usage is lesser Submission time - earlier is lesser Optionally, based on a conditional configuration to enable sizeBasedWeight (default false), an adjustment to boost larger applications (to offset the natural preference for smaller applications) will adjust the resource usage value based on demand, dividing it by the below value: Math.log1p(app memory demand) / Math.log(2); In cases where the above is indeterminate (two applications are equal after this comparison), behavior falls back to comparison based on the application name, which is lexically FIFO for that comparison (first submitted is lesser) was:Implement a Fair SchedulerOrderingPolicy which prefers to allocate to SchedulerProcesses with least current usage, very similar to the FairScheduler's FairSharePolicy. > Implement a Fair SchedulerOrderingPolicy > > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch > > > Implement a Fair Comparator for the Scheduler Comparator Ordering Policy > which prefers to allocate to SchedulerProcesses with least current usage, > very similar to the FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > An implementation of a Scheduler Comparator for use with the Scheduler > Comparator Ordering Policy will be built with the below comparison for > ordering applications for container assignment (ascending) and for preemption > (descending) > Current resource usage - less usage is lesser > Submission time - earlier is lesser > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > name, which is lexically FIFO for that comparison (first submitted is lesser) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3319) Implement a Fair SchedulerOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3319: -- Attachment: YARN-3319.17.patch With support for configuration via the scheduler's config file > Implement a Fair SchedulerOrderingPolicy > > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch > > > Implement a Fair SchedulerOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3319) Implement a Fair SchedulerOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3319: -- Attachment: YARN-3319.14.patch Same as .13, except it should be possible to apply this patch after applying [YARN-3318] 's .14 patch > Implement a Fair SchedulerOrderingPolicy > > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch > > > Implement a Fair SchedulerOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3319) Implement a Fair SchedulerOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3319: -- Attachment: YARN-3319.13.patch Attaching initial/incomplete patch, it depends on the [YARN-3318] patch of the same index - it is just the additional logic specific to Fairness. Major TODO, sizeBasedWeight. > Implement a Fair SchedulerOrderingPolicy > > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch > > > Implement a Fair SchedulerOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
