wilfred-s commented on PR #724: URL: https://github.com/apache/yunikorn-core/pull/724#issuecomment-1827002898
I think I need to answer the questions in the reverse order they were made: > As a first step in decommissioning, Node has been marked schedulable as false. This happens well before all the allocations cleanup, root queue resource adjustments etc. Since we are doing the check is node schedulable in all try*() calls, just wondering how do we encounter this problem between root queue adjustments & allocations cleanup? The problem is not the fact that the node is marked unschedulable or not. The node resources are removed from the root queue max resources. That happens before we remove the allocations from the application(s) and queue(s). So assume the following: - cluster with 2 nodes A and B: each with 32GB - the root queue max is set to 64GB. - both nodes have a usage of 30GB, root queue usage is thus 60GB - root queue has a headroom of 4GB (64GB - 60GB). - queues below the root do not have a max set (simplify the example) Now we're allocating a 2GB allocation. Fits well inside the headroom of the root queue. Allocation is assigned to node A, all fits and looks OK. When the allocation is added to the node and queue is when the issue starts. During the time the scheduler is trying to find a node for the allocation the removal of node B is requested, and partially processed: the root queue max is set to 32GB. The allocations are not yet removed, which means the root queue usage is still 60GB. We now try to update the usage of the queues as part of the node A confirmation. At the root queue the usage _would_ become 60GB + 2GB == 62GB. This fails. The max of the root queue was set to 32GB due to the node removal. _Any_ increment would thus put the root queue over its max. Root queue usage will still be somewhere between 30GB and 60GB. Any allocation from the removed node that is left would cause the root queue to go over its max value as it already is over... After this scheduling will not happen until all allocations from the removed node are removed. The headroom of the root queue will be negative until all is done. It thus only happens once and it fixes itself. We should prevent this from happening anyway. > I think this is related to https://github.com/apache/yunikorn-core/pull/505. I proposed a solution with an extra lock This update is not linked to configuration. Node additions or removals are not config and are not part of the scheduling cycle. The change proposed there does not cover this behaviour. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
