[
https://issues.apache.org/jira/browse/YUNIKORN-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YUNIKORN-3411:
-------------------------------------
Labels: pull-request-available (was: )
> Application write lock held across the RM release round trip
> ------------------------------------------------------------
>
> Key: YUNIKORN-3411
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3411
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - scheduler
> Reporter: Dale Richardson
> Assignee: Andy Huang
> Priority: Major
> Labels: pull-request-available
>
> {{Application.notifyRMAllocationReleased}} hands a release to the RM proxy
> and then waits on an unbuffered channel for the answer. Every path that
> reaches it holds the application write lock across that wait: the state
> timeout ({{timeoutStateTimer}}), placeholder timeout and placeholder
> replacement ({{timeoutPlaceholderProcessing}}, {{tryPlaceholderAllocate}}),
> and the whole preemption chain from {{tryAllocate}} through
> {{Preemptor.TryPreemption}} and the required-node preemptor.
> {code} sa.rmEventHandler.HandleEvent(releaseEvent)
> // Wait from channel
> result := <-c // no timeout; the application lock is held by
> every caller
> {code}
> The RM proxy never takes the application lock, so this is not a deadlock; it
> is an unbounded stall. How long the application is locked is however long the
> shim takes to process the release. It becomes permanent if the RM event queue
> is full and the event is dropped ({{HandleEvent}} drops on a full queue with
> a {{DPanic}} that is only a log line in production), or if the RM proxy
> goroutine is wedged for another reason. On the preemption chain the lock
> belongs to the scheduling goroutine, so all scheduling stops for the
> duration; on the timer paths that one application can neither schedule,
> complete nor be queried. This was measured as a stall during the July review
> with a test that delays the RM reply (not in the tree). YUNIKORN-1099 fixed a
> nil-channel deadlock in the same function; the lock-across-round-trip shape
> stayed.
> Fix: notify after unlocking. The timer paths can collect the releases under
> the lock, unlock, then notify. The preemption chain needs the victim list
> handed back up to {{Queue.TryAllocate}} (or a callback run after
> {{tryAllocate}} releases the lock). Bounding the wait is a fallback, not a
> fix.
> Marker: 11 {{+lockblockingignore}} sites across {{application.go}},
> {{preemption.go}} and {{required_node_preemptor.go}} carry this JIRA; the fix
> removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
