Dale Richardson created YUNIKORN-3411:
-----------------------------------------
Summary: Application write lock held across the RM release round
trip
Key: YUNIKORN-3411
URL: https://issues.apache.org/jira/browse/YUNIKORN-3411
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Reporter: Dale Richardson
{{Application.notifyRMAllocationReleased}} hands a release to the RM proxy and
then waits on an unbuffered channel for the answer. Every path that reaches it
holds the application write lock across that wait: the state timeout
({{timeoutStateTimer}}), placeholder timeout and placeholder replacement
({{timeoutPlaceholderProcessing}}, {{tryPlaceholderAllocate}}), and the whole
preemption chain from {{tryAllocate}} through {{Preemptor.TryPreemption}} and
the required-node preemptor.
{code} sa.rmEventHandler.HandleEvent(releaseEvent)
// Wait from channel
result := <-c // no timeout; the application lock is held by
every caller
{code}
The RM proxy never takes the application lock, so this is not a deadlock; it is
an unbounded stall. How long the application is locked is however long the shim
takes to process the release. It becomes permanent if the RM event queue is
full and the event is dropped ({{HandleEvent}} drops on a full queue with a
{{DPanic}} that is only a log line in production), or if the RM proxy goroutine
is wedged for another reason. On the preemption chain the lock belongs to the
scheduling goroutine, so all scheduling stops for the duration; on the timer
paths that one application can neither schedule, complete nor be queried. This
was measured as a stall during the July review with a test that delays the RM
reply (not in the tree). YUNIKORN-1099 fixed a nil-channel deadlock in the same
function; the lock-across-round-trip shape stayed.
Fix: notify after unlocking. The timer paths can collect the releases under the
lock, unlock, then notify. The preemption chain needs the victim list handed
back up to {{Queue.TryAllocate}} (or a callback run after {{tryAllocate}}
releases the lock). Bounding the wait is a fallback, not a fix.
Marker: 11 {{+lockblockingignore}} sites across {{application.go}},
{{preemption.go}} and {{required_node_preemptor.go}} carry this JIRA; the fix
removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
