[
https://issues.apache.org/jira/browse/YUNIKORN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301368#comment-17301368
]
Wilfred Spiegelenburg commented on YUNIKORN-567:
------------------------------------------------
I walked over the changes in the PR and the call stacks:
PR #256 is not a temporary fix. It complements the removal for the normal
allocations. I do think we're not complete yet with the change. Not sure if you
want to do it from this Jira or log a new one to make it clearer that these are
two separate things.
{{Queue.RemoveApplication()}} was and is the primary way to remove the
application triggered by a removal of the app by the RM. The RM has requested
the application to be removed, the scheduler removes everything related to the
app and confirms that to the shim. In this case we did not account for the
placeholders in the queue. So we have to do that.
We also need to make sure that when the application is removed the timers are
cleaned up correctly. Currently we leave them running until they time out. We
should really stop them when we are finished removing the app.
We have to assume that when the shim says delete the app that is has done its
work. We also set the termination type of the allocation release to
STOPPED_BY_RM. Which means that we do not take any further action on the
response.
This is a different order than a remove and or delete that is instigated by the
core. When we cleanup the application in the {{timeoutPlaceholderProcessing()}}
we have two cases.
* First case we clean up all lingering placeholder allocations on the running
app
* Second case is the fail of the which cleans up lingering asks no response
needed from the shim) and all placeholders after which we fail the app.
The cleanup of the placeholders in both these cases are instigated by the core
and we need to wait for the cleanup to happen on the shim side before we
proceed. It is not like the remove of the app signalled by the RM. This comes
as an unexpected request for the shim not when the app is deleted on the shim
side.
For case 1 we do not have a problem. The placeholders are terminated and the
app runs as per normal and is not moved to Completed until all is finished.
We do NOT have an issue in the states leading to Completed as we have already
handled it there (see below)
For the failure case we immediately unlink the queue as we move into the FAILED
state. As the move calls the {{moveTerminatedApp()}} via the callback. That
causes the issue. This might require a new state to do this in two steps:
trigger the cleanup move to Failing state, when all is cleaned up move to
Failed.
BTW: introducing a new state for Failing should also include the rename of
Waiting to Completing as that is inline with what the state does and lines up
between the two final states.
SIDE NOTE: we might want to distinguish the text in the release of asks and
allocations in the lines 280 and 282. Both asks unallocated and allocated
placeholders will show the same message. Nicer would be to distinguish the two:
{code:java}
sa.notifyRMAllocationAskReleased(sa.rmID, sa.getAllRequests(),
si.TerminationType_TIMEOUT, "releasing placeholders on placeholder timeout")
...
sa.notifyRMAllocationReleased(sa.rmID, sa.getPlaceholderAllocations(),
si.TerminationType_TIMEOUT, "releasing placeholders on placeholder
timeout"){code}
> Queue resources are not cleaned up after placeholder cleanup
> ------------------------------------------------------------
>
> Key: YUNIKORN-567
> URL: https://issues.apache.org/jira/browse/YUNIKORN-567
> Project: Apache YuniKorn
> Issue Type: Bug
> Reporter: Kinga Marton
> Assignee: Kinga Marton
> Priority: Major
> Labels: pull-request-available
>
> When removing an application from the queue only the real allocated resources
> are handled, the resources consumed by the placeholders are not decreased
> from the queue usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
