Ajai Omtri created YARN-6567:
--------------------------------
Summary: Flexible Workload Management
Key: YARN-6567
URL: https://issues.apache.org/jira/browse/YARN-6567
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Reporter: Ajai Omtri
Priority: Minor
Yarn workload management can be little more dynamic.
1. Create yarn pool by specifying more than one Secondary AD group.
Scenario:
In a multi-tenant cluster there can be hundreds of AD groups per tenant and
hundreds of users per AD group. We want a way to group like workloads into
single yarn pool by specifying multiple secondary AD Groups.
Ex: All the ETL workloads of tenants needs to go into one yarn pool. This
requires addition of all ETL related AD groups into one yarn pool.
2. Demotions
Scenario: A particular workload/job has been started in a high priority yarn
pool based on the assumption that it would finish quickly but due to some data
issue/change in the code/query etc. - now it is running longer and consuming
high amounts of resources for long time. In this case we want to demote this to
a lower resource allocated yarn pool. We don’t want this one run-away
workload/job to dominate the cluster because our assumption was wrong.
Ex: If any workload in the yarn pool runs for X minutes and/or consumes Y
resources either alert me or push to another yarn pool. User can keep demoting
and can push to a yarn pool which has capped resources - like Penalty box.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
