[
https://issues.apache.org/jira/browse/YUNIKORN-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3364:
--------------------------------------
Description:
{{EventStreaming.CreateEventStream}} replays the event history to a new
consumer with a bare blocking send ({{{}consumer <- event{}}}) *outside* the
forwarder's select ({{{}event_streaming.go{}}}, the history-replay loop at
stream open). If the requesting client closes the connection before the stream
is really started, and it requested a non-zero history (not the default), that
push blocks with no stop channel reachable: the forwarder goroutine, its
consumer buffer and its {{seen}} map are pinned for the life of the process.
Confirmed by the maintainer on PR #1124.
Proposed fix: make the history-replay send selectable so
{{{}stop{}}}/{{{}e.stopCh{}}} can reach it:
{code:java}
select {
case consumer <- event:
case <-stop:
close(consumer)
return
case <-e.stopCh:
close(consumer)
return
}
{code}
Then delete the {{events.(*EventStreaming).CreateEventStream}} exemption in
{{{}pkg/common/leakcheck/leakcheck.go{}}}.
Note: a second, test-only cause shares the same top frame (a test that creates
a stream and never calls {{{}RemoveStream{}}}); it is tracked separately as the
stream test-hygiene follow-up.
was:
Follow-up to YUNIKORN-3357 (PR #1124); burns down the leakcheck exemption
events.(*EventStreaming).CreateEventStream.func1.
{{EventStreaming.CreateEventStream}} forwards events with bare blocking sends
({{{}consumer <- event{}}} in the history-replay loop and in the select body,
{{event_streaming.go}} ~lines 109 and 127). When a consumer stops reading (a
disconnected or stalled REST event-stream client), the forwarder blocks on the
send *outside* its select and can no longer observe {{stop}} or
{{{}e.stopCh{}}}. The slow-consumer eviction path then closes both channels,
but closing a channel cannot unblock a sender: the goroutine, its 1000-event
consumer buffer and its {{seen}} map are pinned for the life of the process —
exactly the case the eviction logic exists to handle.
Proposed fix: make both sends selectable:
{code:java}
select {
case consumer <- event:
case <-stop:
close(consumer)
return
case <-e.stopCh:
close(consumer)
return
}
{code}
A second, test-only cause shares this top frame (a test that creates a stream
and never calls {{{}RemoveStream{}}}); fix that alongside, then delete the
exemption in {{{}pkg/common/leakcheck/leakcheck.go{}}}.
Summary: Event stream history replay blocks forever if the client
disconnects before the stream opens with a non-zero history request (was:
Event stream forwarder leaks permanently when a consumer stalls; slow-consumer
eviction cannot release it)
> Event stream history replay blocks forever if the client disconnects before
> the stream opens with a non-zero history request
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3364
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3364
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Dale Richardson
> Priority: Minor
> Labels: newbie
>
> {{EventStreaming.CreateEventStream}} replays the event history to a new
> consumer with a bare blocking send ({{{}consumer <- event{}}}) *outside* the
> forwarder's select ({{{}event_streaming.go{}}}, the history-replay loop at
> stream open). If the requesting client closes the connection before the
> stream is really started, and it requested a non-zero history (not the
> default), that push blocks with no stop channel reachable: the forwarder
> goroutine, its consumer buffer and its {{seen}} map are pinned for the life
> of the process. Confirmed by the maintainer on PR #1124.
> Proposed fix: make the history-replay send selectable so
> {{{}stop{}}}/{{{}e.stopCh{}}} can reach it:
> {code:java}
> select {
> case consumer <- event:
> case <-stop:
> close(consumer)
> return
> case <-e.stopCh:
> close(consumer)
> return
> }
> {code}
> Then delete the {{events.(*EventStreaming).CreateEventStream}} exemption in
> {{{}pkg/common/leakcheck/leakcheck.go{}}}.
> Note: a second, test-only cause shares the same top frame (a test that
> creates a stream and never calls {{{}RemoveStream{}}}); it is tracked
> separately as the stream test-hygiene follow-up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
