[
https://issues.apache.org/jira/browse/YUNIKORN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111098#comment-18111098
]
Hedger Lai commented on YUNIKORN-3409:
--------------------------------------
Hi [~tingyao], Thanks!
> AddRejectedApplication writes the rejected-application map without the
> partition lock
> -------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3409
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3409
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - scheduler
> Reporter: Dale Richardson
> Assignee: Hedger Lai
> Priority: Critical
> Labels: pull-request-available
>
> {{PartitionContext.AddRejectedApplication}} creates and writes
> {{pc.rejectedApplications}} with no lock. It runs on the RM event goroutine
> (from {{handleRMUpdateApplicationEvent}}, for both the user-rejected and
> placement-rejected paths, neither of which holds the partition lock). The
> same map is read under {{pc.RLock()}} by the REST handlers
> ({{GetRejectedApplications}}, which the web UI polls) and walked and deleted
> from by the partition manager's expired-app cleanup.
> {code}func (pc *PartitionContext) AddRejectedApplication(rejectedApplication
> *objects.Application, rejectedMessage string) {
> [...]
> if pc.rejectedApplications == nil {
> pc.rejectedApplications = make(map[string]*objects.Application)
> }
> pc.rejectedApplications[rejectedApplication.ApplicationID] =
> rejectedApplication
> }
> {code}
> A rejection landing while the UI is polling is {{fatal error: concurrent map
> read and map write}}, which cannot be recovered: the scheduler process dies.
> This was reproduced during the July concurrency review with a small test
> driving concurrent rejections against a REST reader (it fired within a few
> hundred milliseconds); the test is not in the tree. The lock has been missing
> since the map was introduced (YUNIKORN-1093, v1.0.0). The lazily-created map
> header is also read without the lock in {{getRejectedAppsByState}}, which the
> same fix covers.
> Fix: take {{pc.Lock()}} around the map creation and insert. The application
> state transition just above it locks the application and can stay outside the
> partition lock.
> Marker: two {{+checklocksignore}} sites in {{partition.go}}
> ({{AddRejectedApplication}}, {{getRejectedAppsByState}}) carry this JIRA; the
> fix removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
