[
https://issues.apache.org/jira/browse/YARN-8191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461238#comment-16461238
]
Gergo Repas commented on YARN-8191:
-----------------------------------
[~wilfreds] Thank you for the review! I uploaded an updated patch. My replies
to your comments:
* > The queueMgr can never be null, it is created when the FS get created the
null check is not needed in the AllocationFileLoaderService
** I removed this check
* > You are adding a new method to the FS (hasApplicationAssignedToQueue) to
check if an application is submitted to the queue but the application attempt
has not been created yet. If we have a large number of queues and applications
this count loop could become really expensive. I would suggest that we track
submitted applications in the FSLeafQueue. We can then easily redefine empty as
all 3 types (submit/run/non-run) being 0.
** Good point, I modified the code to according to your comment.
* > The root and root.default queue should not be marked dynamic when they get
created by the queue manager.
** They are actually marked as dynamic=false
* > I do not understand the change around the isEmpty / shutDownIfEmpty some
code is commented out and shutDownIfEmpty is never used
** I submitted this block comment/method by mistake, I removed it.
* > I think we should redefine isEmpty() in the QueueManager to take into
account the submitted apps instead of creating something new.
** Good point, I updated this method.
* > getDynamicQueueNames should rely on the isDynamic flag that is part of the
queue and not calculated.
** getDynamicQueueNames() is being used while we're setting the isDynamic
flags, that's why I could not rely on the flag itself
> Fair scheduler: queue deletion without RM restart
> -------------------------------------------------
>
> Key: YARN-8191
> URL: https://issues.apache.org/jira/browse/YARN-8191
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.0.1
> Reporter: Gergo Repas
> Assignee: Gergo Repas
> Priority: Major
> Attachments: Queue Deletion in Fair Scheduler.pdf,
> YARN-8191.000.patch, YARN-8191.001.patch, YARN-8191.002.patch,
> YARN-8191.003.patch, YARN-8191.004.patch
>
>
> The Fair Scheduler never cleans up queues even if they are deleted in the
> allocation file, or were dynamically created and are never going to be used
> again. Queues always remain in memory which leads to two following issues.
> # Steady fairshares aren’t calculated correctly due to remaining queues
> # WebUI shows deleted queues, which is confusing for users (YARN-4022).
> We want to support proper queue deletion without restarting the Resource
> Manager:
> # Static queues without any entries that are removed from fair-scheduler.xml
> should be deleted from memory.
> # Dynamic queues without any entries should be deleted.
> # RM Web UI should only show the queues defined in the scheduler at that
> point in time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
