Wilfred Spiegelenburg commented on YARN-8191:

The way the return value for {{removeEmptyIncompatibleQueues}} is managed does 
not make the code easier to read/follow. The return value can now mean two 
things. Looking at what we want to do: having 3 return codes using a Boolean 
object is a much better fit (null, true and false). Handling the null object 
depending on the situation would make it easier. Boolean is a slight overhead 
but I think that is ok. Can we merge {{getIncompatibleQueueName}} and 
{{removeEmptyIncompatibleQueues}} into one method? I do not see a specific 
reason to pull the two apart.

When we call {{removePendingIncompatibleQueues}} you first remove the task from 
the list and then call the execute on the task. What happens if the task still 
can not remove the queue? We have removed the task and can no longer track it. 
We should only remove the task if it was successful, otherwise we should leave 
it in the list and try again later.
Looking at the test you wrote for this we do not cover this case.

> Fair scheduler: queue deletion without RM restart
> -------------------------------------------------
>                 Key: YARN-8191
>                 URL: https://issues.apache.org/jira/browse/YARN-8191
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 3.0.1
>            Reporter: Gergo Repas
>            Assignee: Gergo Repas
>            Priority: Major
>         Attachments: Queue Deletion in Fair Scheduler.pdf, 
> YARN-8191.000.patch, YARN-8191.001.patch, YARN-8191.002.patch, 
> YARN-8191.003.patch, YARN-8191.004.patch, YARN-8191.005.patch, 
> YARN-8191.006.patch, YARN-8191.007.patch, YARN-8191.008.patch, 
> YARN-8191.009.patch, YARN-8191.010.patch, YARN-8191.011.patch, 
> YARN-8191.012.patch
> The Fair Scheduler never cleans up queues even if they are deleted in the 
> allocation file, or were dynamically created and are never going to be used 
> again. Queues always remain in memory which leads to two following issues.
>  # Steady fairshares aren’t calculated correctly due to remaining queues
>  # WebUI shows deleted queues, which is confusing for users (YARN-4022).
> We want to support proper queue deletion without restarting the Resource 
> Manager:
>  # Static queues without any entries that are removed from fair-scheduler.xml 
> should be deleted from memory.
>  # Dynamic queues without any entries should be deleted.
>  # RM Web UI should only show the queues defined in the scheduler at that 
> point in time.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to