Eric Chang created KAFKA-20995:
----------------------------------
Summary: [Draft][KIP-1371] Introduce a Consumer Reactor for state
management and event processing
Key: KAFKA-20995
URL: https://issues.apache.org/jira/browse/KAFKA-20995
Project: Kafka
Issue Type: Improvement
Components: consumer
Reporter: Eric Chang
h2. Status
*Draft problem statement for KIP-1371.* The design and implementation are still
being validated. This issue does not commit to a final architecture.
KIP: [KIP-1371: Introduce a Consumer Reactor for State Management and Event
Processing|https://cwiki.apache.org/confluence/spaces/KAFKA/pages/449282795/KIP-1371%2BIntroduce%2Ba%2BConsumer%2BReactor%2Bfor%2BState%2BManagement%2Band%2BEvent%2BProcessing]
Discussion thread: TBD
h2. Problem
The async regular consumer and share consumer already use a background event
loop, request managers, network I/O, cross-thread event queues, futures, and
data buffers. However, the decisions to poll, wait, retry, publish state,
complete work, and wake the application can still be made on separate paths
that observe different parts or versions of consumer state.
This creates four recurring failure shapes:
* *Urgent work without feasible progress.* A local timer can return zero even
though a coordinator is unavailable or an earlier request is still in flight.
The application and background threads may repeatedly poll without producing
work (KAFKA-20253, KAFKA-20426, KAFKA-20970).
* *Ambiguous empty manager results.* An empty result does not state whether
time, a network completion, or another input can enable progress. This can
produce application/background wakeup ping-pong (KAFKA-20854).
* *Publication and wait ordering races.* State, errors, completions, and
wakeups can become visible in different orders, so an application thread may
enter a wait without observing the condition that should release it
(KAFKA-18641, KAFKA-20397).
* *Distributed lifecycle dependencies.* Coordinator discovery, commit,
leave-group, and shutdown progress can be started or stopped by different
components without one final view of the outstanding work (KAFKA-18569,
KAFKA-19357).
These issues do not prove that one call mechanism caused every bug. They show
that consumer-level timing, lifecycle, and application-visible effect decisions
can depend on state distributed across managers and completion paths.
h2. Desired outcome
The design should make the following properties explicit and testable:
* Each request manager retains its mutable state and domain-specific rules.
* Manager results distinguish work available now, finite time-based retry, and
waiting for an enabling input.
* Cross-manager observations are ordered and applied by the state owner against
the state version that produced the request.
* One immutable aggregate timing decision is published before related
application-visible completions, notifications, or wakeups execute.
* Regular and share consumers reuse the execution mechanics without combining
their protocol state or rules.
* The model remains incremental, bounded, and diagnosable; it must not require
another runtime event bus, signal registry, or dynamic dependency graph unless
a concrete missing wake edge proves one necessary.
h2. Scope
This issue tracks the problem definition and design validation for KIP-1371.
The initial migration is internal and is not intended to change the Kafka
protocol, public Consumer or ShareConsumer APIs, callback thread guarantees, or
the runtime thread name.
Implementation work and pull requests should be linked after the draft design
and POC evidence are reviewed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
