Dale Richardson created YUNIKORN-3414:
-----------------------------------------
Summary: Manager reaches into user/group tracker internals without
the tracker lock
Key: YUNIKORN-3414
URL: https://issues.apache.org/jira/browse/YUNIKORN-3414
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Reporter: Dale Richardson
Three places in {{ugm/manager.go}} touch a tracker's internals while holding
only the manager lock. The tracker's own lock, which every other access to
those fields takes, is not held.
The one that matters is {{resetGroupEarlierUsage}}, reached from
{{UpdateConfig}} when a config reload drops a group limit:
{code} appUsersMap := gt.decreaseAllTrackedResourceUsage(hierarchy)
for app, u := range appUsersMap {
ut := m.userTrackers[u]
delete(ut.appGroupTrackers, app) // manager lock
only; ut may be nil
}
{code}
{{setGroupForApp}}, {{getGroupForApp}} and {{hasGroupForApp}} read and write
the same map under the user tracker lock from the scheduling paths, which do
not hold the manager lock, so this is an unsynchronised map delete against
concurrent reads: the fatal concurrent-map class. The nil case is real too: the
user tracker map and the group's application map are cleaned up on independent
paths ({{DecreaseTrackedResource}} with {{removeApp}}, {{ClearUserTrackers}}),
so the user tracker can be gone while the group still lists the application.
Trigger: a reload that removes a group limit while that group's users are being
scheduled. Not reproduced.
The other two are {{GetUserResources}} and {{GetGroupResources}}, which clone
{{queueTracker.resourceUsage}} under the manager read lock. The queue tracker
is documented lock free and relies on its owner's lock. They have no production
callers today (only the stale {{Tracker}} interface names them), so on their
own they would not be worth a ticket; they should get the same treatment while
this is fixed.
Fix: add a locked {{UserTracker.removeGroupForApp}} mirroring
{{setGroupForApp}} with a nil guard (manager-over-tracker is already the
established order), and route the two accessors through the tracker's locked
getters.
Marker: three {{+checklocksignore}} sites in {{manager.go}} carry this JIRA;
the fix removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
