[
https://issues.apache.org/jira/browse/YUNIKORN-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weichen lai reassigned YUNIKORN-3436:
-------------------------------------
Assignee: weichen lai
> Event stream forwarder leaks on slow-consumer eviction: in-loop consumer send
> is not selectable
> -----------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3436
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3436
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Dale Richardson
> Assignee: weichen lai
> Priority: Major
>
> Sibling of YUNIKORN-3364, which covers the history-replay send at stream
> open. The forwarder goroutine started by {{EventStreaming.CreateEventStream}}
> has a second bare blocking send inside its main loop
> ({{pkg/events/event_streaming.go}}, the {{consumer <- event}} after the
> {{seen}} check, currently line 127). It sits inside the {{for}} but outside
> the {{select}}, so once the forwarder is parked on it neither {{stop}} nor
> {{e.stopCh}} can reach it.
> This is reachable in production through the slow-consumer eviction path:
> # A REST event-stream client stops reading; the handler stalls in {{Encode}}
> under its 5s write deadline.
> # The forwarder keeps draining {{local}} into {{consumer}} until {{consumer}}
> (buffer 1000) is full, then blocks on the bare send.
> # {{PublishEvent}} finds {{local}} full, logs "Listener buffer full due to
> potentially slow consumer, removing it" and calls {{removeEventStream}},
> which closes the stream's stop channel. The forwarder cannot observe it.
> # The handler returns; its deferred {{RemoveStream}} is a no-op (already
> removed). Nothing ever drains {{consumer}}.
> Result: one forwarder goroutine plus its two 1000-entry buffers and {{seen}}
> map are pinned for the life of the process, per slow-consumer episode. The
> stream count exposed by {{GetEventStreams}} goes back to zero, so the leak is
> invisible to the API.
> {{TestEventStreaming_SlowConsumer}} reproduces it exactly: it creates a
> stream, publishes 2500 events with no reader, asserts the stream was evicted,
> and {{Close()}} cannot stop the forwarder. This is the one
> {{CreateEventStream.func1}} goroutine that still leaks under goleak after the
> test-hygiene fix in YUNIKORN-3372 landed (#1135), and it is not addressed by
> the fix for YUNIKORN-3364 in #1133, which makes only the history-replay send
> selectable.
> Proposed fix: the same shape as YUNIKORN-3364, applied to the in-loop send:
> {code:go}select {
> case consumer <- event:
> case <-stop:
> close(consumer)
> return
> case <-e.stopCh:
> close(consumer)
> return
> }
> {code}
> Once both sends are selectable, {{TestEventStreaming_SlowConsumer}} passes
> under goleak without an exemption, and the
> {{events.(*EventStreaming).CreateEventStream}} entry in
> {{pkg/common/leakcheck/leakcheck.go}} (#1124) can be deleted.
> Found with goleak while rebasing #1124 (YUNIKORN-3357).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
