[
https://issues.apache.org/jira/browse/YUNIKORN-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KAI-JAY TU reassigned YUNIKORN-3412:
------------------------------------
Assignee: KAI-JAY TU
> Event system handler reads the fields Stop() rewrites; Stop() blocks under
> the write lock
> -----------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3412
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3412
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - common
> Reporter: Dale Richardson
> Assignee: KAI-JAY TU
> Priority: Major
>
> {{EventSystemImpl.Stop()}} takes the write lock and then does two unbuffered
> sends, {{ec.stop <- struct{}{}}} and the publisher's stop, before closing and
> nil-ing {{ec.channel}}. The handler goroutine reads {{ec.stop}} and
> {{ec.channel}} on every loop iteration without any lock.
> Two problems share that seam. The blocking one: the publisher goroutine that
> {{publisher.stop()}} waits for may be inside {{eventPlugin.SendEvent}}, a
> call into the shim with no timeout. While it is, {{Stop()}} holds the write
> lock and every {{AddEvent}} (read lock) queues behind it: the event system is
> dead until the shim returns. The racing one: the handler's unlocked reads of
> the two channel fields against {{Stop()}}'s writes. On master the second is
> masked by the first (the blocking send orders the read, the receive and the
> write), and fixing the send alone makes the race live; the two have to ship
> together. The path is a config reload that toggles {{event.trackingEnabled}}
> ({{reloadConfig -> restart -> Stop}}) and any in-process stop
> (YUNIKORN-3370). The wedge is reproduced by two tests on the fork fix branch,
> which fail on master. YUNIKORN-3336 and 3363 fixed the restart leak and the
> callback registration; this is {{Stop()}} itself.
> Fix: make {{Stop()}} CAS-first, snapshot and nil the fields under the lock,
> keep {{close(ec.channel)}} inside the lock (because {{AddEvent}} sends under
> the read lock), turn the two sends into closes after unlocking, and pass the
> channels into the goroutines as parameters, the way {{CreateEventStream}}
> already does. {{Stop()}} then no longer waits for the goroutines to
> acknowledge. A fix exists on the tigerquoll fork
> ({{fix/event-system-stop-races}}, PR #15 there) and will be filed once this
> has a number.
> Marker: the handler goroutine in {{event_system.go}} carries this JIRA; the
> fix removes it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
