Dale Richardson created YUNIKORN-3413:
-----------------------------------------
Summary: Wildcard limit config read without the manager lock on
scheduling paths
Key: YUNIKORN-3413
URL: https://issues.apache.org/jira/browse/YUNIKORN-3413
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Reporter: Dale Richardson
{{Manager.getUserWildCardLimitsConfig}} reads {{m.userWildCardLimitsConfig}}
with no lock. It is reached from {{newQueueTracker}} through the package-global
manager, which means from {{IncreaseTrackedResource}}, {{Headroom}} and
{{CanRunApp}} on the scheduling paths, holding only a user or group tracker
lock. {{replaceLimitConfigs}} and {{ClearConfigLimits}} replace the whole map
under the manager write lock on every config reload.
Taking the manager lock in place does not work: {{setUserLimits}} reaches the
same constructor while already holding {{m.Lock()}}, and the tracker-locked
callers would take the manager lock inside a tracker lock, the wrong way round.
Confirmed with the race detector on exactly that chain (test on the fork fix
branch). Beyond the race, each child tracker re-reads the global during one
hierarchy walk, so a reload landing mid-walk builds {{root.parent}} with the
old wildcard limit and {{root.parent.child}} with the new one.
Fix: resolve the wildcard config once at each operation's entry and pass it
through tracker construction. Manager-locked callers read the field directly;
tracker-locked callers take one {{RLock}} snapshot at entry; {{UpdateConfig}}
passes the map it is applying. The map is only ever replaced wholesale, never
written element by element, so the reference is a valid snapshot. A fix exists
on the tigerquoll fork ({{fix/ugm-wildcard-config-threading}}, PR #16 there)
and will be filed once this has a number.
Marker: {{getUserWildCardLimitsConfig}} in {{manager.go}} carries this JIRA;
the fix removes it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
