Dale Richardson created YUNIKORN-3365:
-----------------------------------------
Summary: Shutdown race: scheduler allocation notify blocks forever
because the RM proxy stops without draining pending events
Key: YUNIKORN-3365
URL: https://issues.apache.org/jira/browse/YUNIKORN-3365
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Reporter: Dale Richardson
Follow-up to YUNIKORN-3357 (PR #1124); burns down the leakcheck exemption
scheduler.(*ClusterContext).notifyRMNewAllocation.
{{ServiceContext.StopAll}} stops the scheduler before the RM proxy.
{{Scheduler.handleAllocEvent}} can select a queued allocation event over the
just-closed stop channel; the chain {{handleAllocEvent ->
handleRMUpdateAllocationEvent -> processAllocations -> notifyRMNewAllocation}}
then posts to the RM proxy and blocks on an unbuffered reply channel
({{context.go:803}}), but {{rmproxy.handleRMEvents}} returns on its own stop
without draining {{pendingRMEvents}}, so the reply never arrives and the
goroutine is stuck permanently.
Reproduced in ~43% of full {{pkg/scheduler/tests}} runs (6/14):
{code}
go test -race -count=14 ./pkg/scheduler/tests/
{code}
{{notifyRMAllocationReleased}} has the same unbuffered-reply shape and would
leak under a different top frame; it has not been observed yet, so it is
deliberately not exempted — but the fix should cover it too.
Proposed fix: drain or explicitly abandon in-flight replies on shutdown (or
invert the stop order), then delete the exemption in
{{pkg/common/leakcheck/leakcheck.go}}. Argue both the deadlock and
data-integrity directions preemptively (cf. the YUNIKORN-2629 -> YUNIKORN-2910
regression).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
