Lianet Magrans created KAFKA-20429:
--------------------------------------
Summary: Review async consumer single background queue design
Key: KAFKA-20429
URL: https://issues.apache.org/jira/browse/KAFKA-20429
Project: Kafka
Issue Type: Task
Components: clients, consumer
Reporter: Lianet Magrans
The async consumer relies on a single queue to pass events from the background
to the app thread. But this queue mixes different kinds of events (coord
errors, metadata errors, callbacks, internal updates events like
applyAssignment).
This makes it problematic and complicated to use. When
processingBackgroundEvents from the app thread, we need to filter-out events
depending on what we're processing background events for. E.g., :
* unsubscribe cares about callbacks only, needs to ignore all others (error
prone as new events are added, see
https://issues.apache.org/jira/browse/KAFKA-20428)
* poll cares about all, but actually could maybe benefit from the ability to
process errors and callbacks at different points of the poll loop (see
https://issues.apache.org/jira/browse/KAFKA-20397)
I wonder if re-designing the background events queue would simplify it's usage
from the AsyncKafkaConsumer, and avoid continuously falling into errors when
new events are added. Also we could have a simpler/clearer intention from the
app thread, consuming from each func what it needs (vs consume all and filter
out).
One option that comes to mind is to have separate queues for errors, callbacks,
internal ops. So app thread should be simpler (explicit intention when
processing events from a queue), but to review in detail to see if we could get
a simpler design, less error prone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
