[
https://issues.apache.org/jira/browse/YUNIKORN-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108769#comment-18108769
]
Wilfred Spiegelenburg edited comment on YUNIKORN-3430 at 8/27/26 10:14 AM:
---------------------------------------------------------------------------
Be really careful with this one: the split was made on purpose as the register
into the core does a callback. Node registrations and updates are still
serialised on the events. Changes will thus be applied in the expected order.
Other updates, like pod changes, can happen while the node updates run without
any impact.
{quote}and the wait has no timeout (YUNIKORN-3239 saw it last 53 minutes).
{quote}
The wait time was due to the serialised processing of application and node
events. The long duration was before all the changes went in to unblock the
event loop. The application events were holding back the node events from
getting processed. The application could not progress due to a large app
failing 1000s of allocations each cycle. The application was failing
allocations as it was waiting for node registrations and or other allocations
stopping. All of them were held up by the scheduling cycle processing that
large app. The large app was the highest priority and always the first queue to
be hit. So even when a single allocation was done the large app was checked
again and again blocking proper progress.
None of that can happen anymore due to two changes made YUNIKORN-3115 and
YUNIKORN-3229
was (Author: wifreds):
Be really careful with this one: the split was made on purpose as the register
into the core does a callback. Node registrations and updates are still
serialised on the events. Changes will thus be applied in the expected order.
Other updates, like pod changes, can happen while the node updates run without
any impact.
{quote}and the wait has no timeout (YUNIKORN-3239 saw it last 53 minutes).
{quote}
The wait time was due to the serialised processing of application and node
events. The long duration was before all the changes went in to unblock the
event loop. The application events were holding back the node events from
getting processed. The node application could not progress due to a large app
failing 1000s of allocations each cycle. The application was of failing
allocations as it was waiting for node registrations and or other allocations
stopping. The app was the highest priority and always the first queue to be hit.
None of that can happen anymore due to two changes made YUNIKORN-3115 and
YUNIKORN-3229
> registerNodesInternal releases and retakes the context lock around the wait
> group
> ---------------------------------------------------------------------------------
>
> Key: YUNIKORN-3430
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3430
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Assignee: jimmycmlin
> Priority: Minor
>
> {{Context.registerNodesInternal}} is documented as requiring the context
> write lock, yet it unlocks and re-locks {{ctx.lock}} around its wait for the
> core's node-accepted events, splitting the caller's critical section in two.
> {{updateNodeInternal}} continues after the call with adopted-pod registration
> and {{enableNode}} on state it established before the gap, and the wait has
> no timeout (YUNIKORN-3239 saw it last 53 minutes).
> No corruption has been shown; the hazard is that nothing re-validates the
> invariants after the wait, and the analyser has to exclude the whole function
> from checking.
> Fix: build the request under the lock, release, send and wait, then re-take
> the lock to apply the results, instead of unlocking inside the callee.
> Marker: {{registerNodesInternal}} in {{context.go}} carries a function-level
> {{+checklocksignore}} for this JIRA; the fix removes it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
