[
https://issues.apache.org/jira/browse/YUNIKORN-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Huang reassigned YUNIKORN-3414:
------------------------------------
Assignee: (was: Andy Huang)
> Manager reaches into user/group tracker internals without the tracker lock
> --------------------------------------------------------------------------
>
> Key: YUNIKORN-3414
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3414
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - scheduler
> Reporter: Dale Richardson
> Priority: Major
>
> Three places in {{ugm/manager.go}} touch a tracker's internals while holding
> only the manager lock. The tracker's own lock, which every other access to
> those fields takes, is not held.
> The one that matters is {{resetGroupEarlierUsage}}, reached from
> {{UpdateConfig}} when a config reload drops a group limit:
> {code} appUsersMap :=
> gt.decreaseAllTrackedResourceUsage(hierarchy)
> for app, u := range appUsersMap {
> ut := m.userTrackers[u]
> delete(ut.appGroupTrackers, app) // manager lock
> only; ut may be nil
> }
> {code}
> {{setGroupForApp}}, {{getGroupForApp}} and {{hasGroupForApp}} read and write
> the same map under the user tracker lock from the scheduling paths, which do
> not hold the manager lock, so this is an unsynchronised map delete against
> concurrent reads: the fatal concurrent-map class. The nil case is real too:
> the user tracker map and the group's application map are cleaned up on
> independent paths ({{DecreaseTrackedResource}} with {{removeApp}},
> {{ClearUserTrackers}}), so the user tracker can be gone while the group still
> lists the application. Trigger: a reload that removes a group limit while
> that group's users are being scheduled. Not reproduced.
> The other two are {{GetUserResources}} and {{GetGroupResources}}, which clone
> {{queueTracker.resourceUsage}} under the manager read lock. The queue tracker
> is documented lock free and relies on its owner's lock. They have no
> production callers today (only the stale {{Tracker}} interface names them),
> so on their own they would not be worth a ticket; they should get the same
> treatment while this is fixed.
> Fix: add a locked {{UserTracker.removeGroupForApp}} mirroring
> {{setGroupForApp}} with a nil guard (manager-over-tracker is already the
> established order), and route the two accessors through the tracker's locked
> getters.
> Marker: three {{+checklocksignore}} sites in {{manager.go}} carry this JIRA;
> the fix removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
