[
https://issues.apache.org/jira/browse/KAFKA-20995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Chang updated KAFKA-20995:
-------------------------------
Summary: KIP-1371: Formalize Consumer Reactor Cross-Manager Coordination
and Publication (was: [Draft][KIP-1371] Introduce a Consumer Reactor for state
management and event processing)
> KIP-1371: Formalize Consumer Reactor Cross-Manager Coordination and
> Publication
> -------------------------------------------------------------------------------
>
> Key: KAFKA-20995
> URL: https://issues.apache.org/jira/browse/KAFKA-20995
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Reporter: Eric Chang
> Assignee: Eric Chang
> Priority: Major
>
> h2. Status
> *Draft problem statement for KIP-1371.* The design and implementation are
> still being validated. This issue does not commit to a final architecture.
> KIP: [KIP-1371: Introduce a Consumer Reactor for State Management and Event
> Processing|https://cwiki.apache.org/confluence/spaces/KAFKA/pages/449282795/KIP-1371%2BIntroduce%2Ba%2BConsumer%2BReactor%2Bfor%2BState%2BManagement%2Band%2BEvent%2BProcessing]
> Discussion thread: TBD
> h2. Problem
> A request manager may be in one of three next-poll conditions: it can produce
> work now, passage of time may make work possible, or an external input must
> arrive first. The current deadline-based result records only when another
> poll may occur. For an empty result, it does not identify which condition
> applies.
> The async regular and share consumers already use a background event loop and
> request managers. Waiting, wakeup, completion, and publication decisions can
> still be made on separate paths using incomplete or differently timed views
> of consumer state. This creates four recurring failure shapes:
> * *Urgent work without progress.* A timer reaches zero while coordinator
> availability, assignment, or in-flight state prevents a request from being
> created. This can repeatedly poll the application and background threads
> without producing work
> ([KAFKA-20253|https://issues.apache.org/jira/browse/KAFKA-20253],
> [KAFKA-20426|https://issues.apache.org/jira/browse/KAFKA-20426];
> [KAFKA-20970|https://issues.apache.org/jira/browse/KAFKA-20970] remains
> proposed evidence).
> * *Ambiguous empty manager results.* An empty result does not identify
> whether time or an input can enable progress. A wakeup selected without
> corresponding progress can create application/background ping-pong
> ([KAFKA-20854|https://issues.apache.org/jira/browse/KAFKA-20854]).
> * *Publication and wait ordering races.* State, errors, completions, and
> wakeups can become visible in different orders
> ([KAFKA-18641|https://issues.apache.org/jira/browse/KAFKA-18641];
> [KAFKA-20397|https://issues.apache.org/jira/browse/KAFKA-20397] remains
> proposed evidence).
> * *Distributed lifecycle dependencies.* Coordinator discovery, commit,
> leave-group, and shutdown progress can be started or stopped by different
> components without one final view of outstanding work
> ([KAFKA-18569|https://issues.apache.org/jira/browse/KAFKA-18569],
> [KAFKA-19357|https://issues.apache.org/jira/browse/KAFKA-19357]).
> These issues do not prove that one call mechanism caused every bug. They
> demonstrate that consumer-level timing, ordering, and application-visible
> effects may depend on state distributed across managers and completion paths.
> h2. Desired outcome
> KIP-1371 makes the following behavior explicit and testable:
> * Each request manager retains its mutable state and domain-specific rules.
> * One {{PollResult}} reports produced {{NetworkCommand}} and {{ManagerEvent}}
> values together with one typed {{NextPollCondition}}: immediate progress,
> finite time-driven retry, or input-driven wait.
> * Cross-manager observations are ordered and applied by one state owner
> against the version used to create the request.
> * {{ConsumerReactor}} publishes one immutable aggregate timing decision
> before executing corresponding application-visible completions,
> notifications, or wakeups.
> * Regular, share, and Streams consumers reuse the execution mechanics while
> retaining separate protocol rules.
> * Migration remains phased, runnable, bounded, and diagnosable.
> h2. Scope and POC
> The initial migration is internal. It preserves Kafka protocols, public
> {{Consumer}} and {{ShareConsumer}} APIs, callback thread guarantees, runtime
> thread names, and the existing application/background thread topology.
> POC branch:
> [codex/async-consumer-reactor-poc|https://github.com/unknowntpo/kafka/tree/codex/async-consumer-reactor-poc]
> Implementation phase map:
> [consumer-reactor-poc-phase-map.md|https://github.com/unknowntpo/kafka/blob/cb7e5c4cb7619084d2e91f7efa494b35b1539d09/docs/design/consumer-reactor-poc-phase-map.md]
> The Confluence KIP is the authoritative design. POC class placement and
> compatibility adapters are implementation evidence, not part of the community
> API commitment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
