[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266391#comment-17266391
]
Wangda Tan commented on YARN-10532:
-----------------------------------
[~zhuqi], thanks for the patch, I took a brief look (nothing in detail yet). I
think here're my overall thoughts:
1) Instead of changing GuaranteedOrZeroCapacityOverTimePolicy, I suggest
creating a new Policy (maybe we can make it runnable by default so we don't
have to create another config). The policy just simply monitor queue last used
time and delete queues when needed.
2) The latest patch only remove queue when reinitialize queue is called, it is
not a frequent used method, we have cluster which doesn't have queue
reinitialize called for a long period of time. Can we just call it from the
policy (then we may need to introduce a method in CapacityScheduler to delete
queue).
3) When we delete a queue, we need to check inside Scheduler to make sure 1)
there's nothing running in the queue; 2) last usage timestamp is expired. We
need to do this to avoid racing condition that Policy think a queue is
deletable but Scheduler doesn't.
4) Policy modifies "expiredQueue" field, I suggest to make Policy just read the
state, and let Scheduler delete it.
5) An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word, deleting an
dynamic queue should NEVER fail a submitted application.{code}
> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is
> not being used
> --------------------------------------------------------------------------------------------
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: zhuqi
> Priority: Major
> Attachments: YARN-10532.001.patch
>
>
> It's better if we can delete auto-created queues when they are not in use for
> a period of time (like 5 mins). It will be helpful when we have a large
> number of auto-created queues (e.g. from 500 users), but only a small subset
> of queues are actively used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
