[
https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319929#comment-17319929
]
VADAGA ANANYO RAO commented on YARN-10559:
------------------------------------------
[~epayne], sorry for seeing your comment so late. We have tested this feature
with following configs:
Queue Properties:
'yarn.scheduler.capacity.<queue_name>.ordering-policy': 'fair'
Scheduler configurations:
'yarn.resourcemanager.scheduler.monitor.enable': 'true',
'yarn.resourcemanager.scheduler.monitor.policies' :
'org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy',
'yarn.resourcemanager.monitor.capacity.preemption.intra-queue-preemption.enabled':
'true'
Post this, we submit job1 from user1 to a leaf queue. When job1 completely uses
up the queue capacity, we trigger job2 from user1 to the same leaf queue. We
can observe preemption kicking in for job2 from job1.
I am not sure of the exact error you are facing. If you can provide some more
details of the problems you are facing, I can try and help out with it.
Thank you :)
> Fair sharing intra-queue preemption support in Capacity Scheduler
> -----------------------------------------------------------------
>
> Key: YARN-10559
> URL: https://issues.apache.org/jira/browse/YARN-10559
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Affects Versions: 3.1.4
> Reporter: VADAGA ANANYO RAO
> Assignee: VADAGA ANANYO RAO
> Priority: Major
> Attachments: FairOP_preemption-design_doc_v1.pdf,
> FairOP_preemption-design_doc_v2.pdf, YARN-10559.0001.patch,
> YARN-10559.0002.patch, YARN-10559.0003.patch, YARN-10559.0004.patch,
> YARN-10559.0005.patch, YARN-10559.0006.patch, YARN-10559.0007.patch,
> YARN-10559.0008.patch, YARN-10559.0009.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Usecase:
> Due to the way Capacity Scheduler preemption works, If a single user submits
> a large application to a queue (using 100% of resources), that job will not
> be preempted by future applications from the same user within the same queue.
> This implies that the later applications will be forced to wait for
> completion of the long running application. This prevents multiple long
> running, large, applications from running concurrently.
> Support fair sharing among apps while preempting applications from same queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
