Dale Richardson created YUNIKORN-3422:
-----------------------------------------
Summary: SchedulerCache node-list getters populate their cache
under the read lock
Key: YUNIKORN-3422
URL: https://issues.apache.org/jira/browse/YUNIKORN-3422
Project: Apache YuniKorn
Issue Type: Bug
Components: shim - kubernetes
Reporter: Dale Richardson
{{SchedulerCache.GetNodesInfo}}, {{GetNodesInfoPodsWithAffinity}} and
{{GetNodesInfoPodsWithReqAntiAffinity}} lazily build their list and assign it
to the cache field the first time they are called after an invalidation. Their
only callers are the predicate shared lister, reached from {{IsPodFitNode}} /
{{IsPodFitNodeViaPreemption}} under {{LockForReads}}, i.e. the read lock.
{code}func (cache *SchedulerCache) GetNodesInfo() []fwk.NodeInfo {
if cache.nodesInfo == nil {
nodeList := make([]fwk.NodeInfo, 0, len(cache.nodesMap))
[...]
cache.nodesInfo = nodeList // written under the read lock
}
return cache.nodesInfo
}
{code}
The core's preemption runs predicate checks on up to ten goroutines at once
({{preemptCheckConcurrency}}), so two of them regularly see {{nil}} together
and both write. Confirmed with the race detector on that path during the July
review (no test in the tree). Concurrent writes of a slice header can publish a
torn one, which would panic inside PreFilter; that consequence is traced, not
observed. Invalidation ({{nodesInfo = nil}}) is already under the write lock;
only the population is wrong.
Fix: build the three lists at invalidation time, under the write lock already
held there, or publish them through an {{atomic.Pointer}}. Fix all three
getters together.
Marker: three {{+checklocksignore}} sites in {{scheduler_cache.go}} carry this
JIRA; the fix removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
