Dale Richardson created YUNIKORN-3409:
-----------------------------------------
Summary: AddRejectedApplication writes the rejected-application
map without the partition lock
Key: YUNIKORN-3409
URL: https://issues.apache.org/jira/browse/YUNIKORN-3409
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Reporter: Dale Richardson
{{PartitionContext.AddRejectedApplication}} creates and writes
{{pc.rejectedApplications}} with no lock. It runs on the RM event goroutine
(from {{handleRMUpdateApplicationEvent}}, for both the user-rejected and
placement-rejected paths, neither of which holds the partition lock). The same
map is read under {{pc.RLock()}} by the REST handlers
({{GetRejectedApplications}}, which the web UI polls) and walked and deleted
from by the partition manager's expired-app cleanup.
{code}func (pc *PartitionContext) AddRejectedApplication(rejectedApplication
*objects.Application, rejectedMessage string) {
[...]
if pc.rejectedApplications == nil {
pc.rejectedApplications = make(map[string]*objects.Application)
}
pc.rejectedApplications[rejectedApplication.ApplicationID] =
rejectedApplication
}
{code}
A rejection landing while the UI is polling is {{fatal error: concurrent map
read and map write}}, which cannot be recovered: the scheduler process dies.
This was reproduced during the July concurrency review with a small test
driving concurrent rejections against a REST reader (it fired within a few
hundred milliseconds); the test is not in the tree. The lock has been missing
since the map was introduced (YUNIKORN-1093, v1.0.0). The lazily-created map
header is also read without the lock in {{getRejectedAppsByState}}, which the
same fix covers.
Fix: take {{pc.Lock()}} around the map creation and insert. The application
state transition just above it locks the application and can stay outside the
partition lock.
Marker: two {{+checklocksignore}} sites in {{partition.go}}
({{AddRejectedApplication}}, {{getRejectedAppsByState}}) carry this JIRA; the
fix removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
