[
https://issues.apache.org/jira/browse/KAFKA-20995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Chang reassigned KAFKA-20995:
----------------------------------
Assignee: Eric Chang
> [Draft][KIP-1371] Introduce a Consumer Reactor for state management and event
> processing
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-20995
> URL: https://issues.apache.org/jira/browse/KAFKA-20995
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Reporter: Eric Chang
> Assignee: Eric Chang
> Priority: Major
>
> h2. Status
> *Draft problem statement for KIP-1371.* The design and implementation are
> still being validated. This issue does not commit to a final architecture.
> KIP: [KIP-1371: Introduce a Consumer Reactor for State Management and Event
> Processing|https://cwiki.apache.org/confluence/spaces/KAFKA/pages/449282795/KIP-1371%2BIntroduce%2Ba%2BConsumer%2BReactor%2Bfor%2BState%2BManagement%2Band%2BEvent%2BProcessing]
> Discussion thread: TBD
> h2. Problem
> The async regular consumer and share consumer already use a background event
> loop, request managers, network I/O, cross-thread event queues, futures, and
> data buffers. However, the decisions to poll, wait, retry, publish state,
> complete work, and wake the application can still be made on separate paths
> that observe different parts or versions of consumer state.
> This creates four recurring failure shapes:
> * *Urgent work without feasible progress.* A local timer can return zero even
> though a coordinator is unavailable or an earlier request is still in flight.
> The application and background threads may repeatedly poll without producing
> work (KAFKA-20253, KAFKA-20426, KAFKA-20970).
> * *Ambiguous empty manager results.* An empty result does not state whether
> time, a network completion, or another input can enable progress. This can
> produce application/background wakeup ping-pong (KAFKA-20854).
> * *Publication and wait ordering races.* State, errors, completions, and
> wakeups can become visible in different orders, so an application thread may
> enter a wait without observing the condition that should release it
> (KAFKA-18641, KAFKA-20397).
> * *Distributed lifecycle dependencies.* Coordinator discovery, commit,
> leave-group, and shutdown progress can be started or stopped by different
> components without one final view of the outstanding work (KAFKA-18569,
> KAFKA-19357).
> These issues do not prove that one call mechanism caused every bug. They show
> that consumer-level timing, lifecycle, and application-visible effect
> decisions can depend on state distributed across managers and completion
> paths.
> h2. Desired outcome
> The design should make the following properties explicit and testable:
> * Each request manager retains its mutable state and domain-specific rules.
> * Manager results distinguish work available now, finite time-based retry,
> and waiting for an enabling input.
> * Cross-manager observations are ordered and applied by the state owner
> against the state version that produced the request.
> * One immutable aggregate timing decision is published before related
> application-visible completions, notifications, or wakeups execute.
> * Regular and share consumers reuse the execution mechanics without combining
> their protocol state or rules.
> * The model remains incremental, bounded, and diagnosable; it must not
> require another runtime event bus, signal registry, or dynamic dependency
> graph unless a concrete missing wake edge proves one necessary.
> h2. Scope
> This issue tracks the problem definition and design validation for KIP-1371.
> The initial migration is internal and is not intended to change the Kafka
> protocol, public Consumer or ShareConsumer APIs, callback thread guarantees,
> or the runtime thread name.
> Implementation work and pull requests should be linked after the draft design
> and POC evidence are reviewed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
