Dale Richardson created YUNIKORN-3427:
-----------------------------------------
Summary: flushReleaseableTasks removes from the context map and
reads task state under the application lock only
Key: YUNIKORN-3427
URL: https://issues.apache.org/jira/browse/YUNIKORN-3427
Project: Apache YuniKorn
Issue Type: Bug
Components: shim - kubernetes
Reporter: Dale Richardson
{{Application.flushReleaseableTasks}} runs as the enter-Accepted callback under
{{app.handle}}'s write lock and nothing else. On the all-terminated path it
calls {{app.context.removeApplication}}, the internal variant that deletes from
{{ctx.applications}} with no context lock, while {{GetAllApplications}} reads
that map under {{ctx.RLock()}} on every scheduling tick. On the other path it
calls {{task.releaseAllocation(true)}} for each deferred task, reading
{{terminationType}}, {{allocationKey}} and friends that {{SetTaskPod}} and the
rollback paths write under {{task.lock}}.
The context-map delete needs an application whose deferred tasks are all
terminated by the time the core accepts it (exactly the YUNIKORN-3089 scenario)
colliding with the ticker's read: a concurrent map read and write, i.e. fatal.
Traced, not reproduced. The fix for the linked {{shouldAppRelease}} deadlock
deliberately leaves both of these out of scope.
Fix: take {{task.lock}} inside the loop (application before task is the
established order), and route the removal through {{Context.RemoveApplication}}
from outside the application lock, or defer it to the caller.
Marker: two {{+checklocksignore}} sites in {{flushReleaseableTasks}} carry this
JIRA; the fix removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
