[
https://issues.apache.org/jira/browse/KAFKA-20995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Chang updated KAFKA-20995:
-------------------------------
Description:
h2. Status
*Draft problem statement for KIP-1371.* The design and implementation are still
being validated. This issue does not commit to a final architecture.
KIP: [KIP-1371: Introduce a Consumer Reactor for State Management and Event
Processing|https://cwiki.apache.org/confluence/spaces/KAFKA/pages/449282795/KIP-1371%2BIntroduce%2Ba%2BConsumer%2BReactor%2Bfor%2BState%2BManagement%2Band%2BEvent%2BProcessing]
Discussion thread: TBD
h2. Problem
A request manager may be in one of three next-poll conditions: it can produce
work now, passage of time may make work possible, or an external input must
arrive first. The current deadline-based result records only when another poll
may occur. For an empty result, it does not identify which condition applies.
The async regular and share consumers already use a background event loop and
request managers. Waiting, wakeup, completion, and publication decisions can
still be made on separate paths using incomplete or differently timed views of
consumer state. This creates four recurring failure shapes:
* *Urgent work without progress.* A timer reaches zero while coordinator
availability, assignment, or in-flight state prevents a request from being
created. This can repeatedly poll the application and background threads
without producing work
([KAFKA-20253|https://issues.apache.org/jira/browse/KAFKA-20253],
[KAFKA-20426|https://issues.apache.org/jira/browse/KAFKA-20426];
[KAFKA-20970|https://issues.apache.org/jira/browse/KAFKA-20970] remains
proposed evidence).
* *Ambiguous empty manager results.* An empty result does not identify whether
time or an input can enable progress. A wakeup selected without corresponding
progress can create application/background ping-pong
([KAFKA-20854|https://issues.apache.org/jira/browse/KAFKA-20854]).
* *Publication and wait ordering races.* State, errors, completions, and
wakeups can become visible in different orders
([KAFKA-18641|https://issues.apache.org/jira/browse/KAFKA-18641];
[KAFKA-20397|https://issues.apache.org/jira/browse/KAFKA-20397] remains
proposed evidence).
* *Distributed lifecycle dependencies.* Coordinator discovery, commit,
leave-group, and shutdown progress can be started or stopped by different
components without one final view of outstanding work
([KAFKA-18569|https://issues.apache.org/jira/browse/KAFKA-18569],
[KAFKA-19357|https://issues.apache.org/jira/browse/KAFKA-19357]).
These issues do not prove that one call mechanism caused every bug. They
demonstrate that consumer-level timing, ordering, and application-visible
effects may depend on state distributed across managers and completion paths.
h2. Desired outcome
KIP-1371 makes the following behavior explicit and testable:
* Each request manager retains its mutable state and domain-specific rules.
* One {{PollResult}} reports produced {{NetworkCommand}} and {{ManagerEvent}}
values together with one typed {{NextPollCondition}}: immediate progress,
finite time-driven retry, or input-driven wait.
* Cross-manager observations are ordered and applied by one state owner against
the version used to create the request.
* {{ConsumerReactor}} publishes one immutable aggregate timing decision before
executing corresponding application-visible completions, notifications, or
wakeups.
* Regular, share, and Streams consumers reuse the execution mechanics while
retaining separate protocol rules.
* Migration remains phased, runnable, bounded, and diagnosable.
h2. Scope and POC
The initial migration is internal. It preserves Kafka protocols, public
{{Consumer}} and {{ShareConsumer}} APIs, callback thread guarantees, runtime
thread names, and the existing application/background thread topology.
POC branch:
[codex/async-consumer-reactor-poc|https://github.com/unknowntpo/kafka/tree/codex/async-consumer-reactor-poc]
Implementation phase map:
[consumer-reactor-poc-phase-map.md|https://github.com/unknowntpo/kafka/blob/cb7e5c4cb7619084d2e91f7efa494b35b1539d09/docs/design/consumer-reactor-poc-phase-map.md]
The Confluence KIP is the authoritative design. POC class placement and
compatibility adapters are implementation evidence, not part of the community
API commitment.
was:
h2. Status
*Draft problem statement for KIP-1371.* The design and implementation are still
being validated. This issue does not commit to a final architecture.
KIP: [KIP-1371: Introduce a Consumer Reactor for State Management and Event
Processing|https://cwiki.apache.org/confluence/spaces/KAFKA/pages/449282795/KIP-1371%2BIntroduce%2Ba%2BConsumer%2BReactor%2Bfor%2BState%2BManagement%2Band%2BEvent%2BProcessing]
Discussion thread: TBD
h2. Problem
The async regular consumer and share consumer already use a background event
loop, request managers, network I/O, cross-thread event queues, futures, and
data buffers. However, the decisions to poll, wait, retry, publish state,
complete work, and wake the application can still be made on separate paths
that observe different parts or versions of consumer state.
This creates four recurring failure shapes:
* *Urgent work without feasible progress.* A local timer can return zero even
though a coordinator is unavailable or an earlier request is still in flight.
The application and background threads may repeatedly poll without producing
work (KAFKA-20253, KAFKA-20426, KAFKA-20970).
* *Ambiguous empty manager results.* An empty result does not state whether
time, a network completion, or another input can enable progress. This can
produce application/background wakeup ping-pong (KAFKA-20854).
* *Publication and wait ordering races.* State, errors, completions, and
wakeups can become visible in different orders, so an application thread may
enter a wait without observing the condition that should release it
(KAFKA-18641, KAFKA-20397).
* *Distributed lifecycle dependencies.* Coordinator discovery, commit,
leave-group, and shutdown progress can be started or stopped by different
components without one final view of the outstanding work (KAFKA-18569,
KAFKA-19357).
These issues do not prove that one call mechanism caused every bug. They show
that consumer-level timing, lifecycle, and application-visible effect decisions
can depend on state distributed across managers and completion paths.
h2. Desired outcome
The design should make the following properties explicit and testable:
* Each request manager retains its mutable state and domain-specific rules.
* Manager results distinguish work available now, finite time-based retry, and
waiting for an enabling input.
* Cross-manager observations are ordered and applied by the state owner against
the state version that produced the request.
* One immutable aggregate timing decision is published before related
application-visible completions, notifications, or wakeups execute.
* Regular and share consumers reuse the execution mechanics without combining
their protocol state or rules.
* The model remains incremental, bounded, and diagnosable; it must not require
another runtime event bus, signal registry, or dynamic dependency graph unless
a concrete missing wake edge proves one necessary.
h2. Scope
This issue tracks the problem definition and design validation for KIP-1371.
The initial migration is internal and is not intended to change the Kafka
protocol, public Consumer or ShareConsumer APIs, callback thread guarantees, or
the runtime thread name.
Implementation work and pull requests should be linked after the draft design
and POC evidence are reviewed.
> [Draft][KIP-1371] Introduce a Consumer Reactor for state management and event
> processing
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-20995
> URL: https://issues.apache.org/jira/browse/KAFKA-20995
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Reporter: Eric Chang
> Assignee: Eric Chang
> Priority: Major
>
> h2. Status
> *Draft problem statement for KIP-1371.* The design and implementation are
> still being validated. This issue does not commit to a final architecture.
> KIP: [KIP-1371: Introduce a Consumer Reactor for State Management and Event
> Processing|https://cwiki.apache.org/confluence/spaces/KAFKA/pages/449282795/KIP-1371%2BIntroduce%2Ba%2BConsumer%2BReactor%2Bfor%2BState%2BManagement%2Band%2BEvent%2BProcessing]
> Discussion thread: TBD
> h2. Problem
> A request manager may be in one of three next-poll conditions: it can produce
> work now, passage of time may make work possible, or an external input must
> arrive first. The current deadline-based result records only when another
> poll may occur. For an empty result, it does not identify which condition
> applies.
> The async regular and share consumers already use a background event loop and
> request managers. Waiting, wakeup, completion, and publication decisions can
> still be made on separate paths using incomplete or differently timed views
> of consumer state. This creates four recurring failure shapes:
> * *Urgent work without progress.* A timer reaches zero while coordinator
> availability, assignment, or in-flight state prevents a request from being
> created. This can repeatedly poll the application and background threads
> without producing work
> ([KAFKA-20253|https://issues.apache.org/jira/browse/KAFKA-20253],
> [KAFKA-20426|https://issues.apache.org/jira/browse/KAFKA-20426];
> [KAFKA-20970|https://issues.apache.org/jira/browse/KAFKA-20970] remains
> proposed evidence).
> * *Ambiguous empty manager results.* An empty result does not identify
> whether time or an input can enable progress. A wakeup selected without
> corresponding progress can create application/background ping-pong
> ([KAFKA-20854|https://issues.apache.org/jira/browse/KAFKA-20854]).
> * *Publication and wait ordering races.* State, errors, completions, and
> wakeups can become visible in different orders
> ([KAFKA-18641|https://issues.apache.org/jira/browse/KAFKA-18641];
> [KAFKA-20397|https://issues.apache.org/jira/browse/KAFKA-20397] remains
> proposed evidence).
> * *Distributed lifecycle dependencies.* Coordinator discovery, commit,
> leave-group, and shutdown progress can be started or stopped by different
> components without one final view of outstanding work
> ([KAFKA-18569|https://issues.apache.org/jira/browse/KAFKA-18569],
> [KAFKA-19357|https://issues.apache.org/jira/browse/KAFKA-19357]).
> These issues do not prove that one call mechanism caused every bug. They
> demonstrate that consumer-level timing, ordering, and application-visible
> effects may depend on state distributed across managers and completion paths.
> h2. Desired outcome
> KIP-1371 makes the following behavior explicit and testable:
> * Each request manager retains its mutable state and domain-specific rules.
> * One {{PollResult}} reports produced {{NetworkCommand}} and {{ManagerEvent}}
> values together with one typed {{NextPollCondition}}: immediate progress,
> finite time-driven retry, or input-driven wait.
> * Cross-manager observations are ordered and applied by one state owner
> against the version used to create the request.
> * {{ConsumerReactor}} publishes one immutable aggregate timing decision
> before executing corresponding application-visible completions,
> notifications, or wakeups.
> * Regular, share, and Streams consumers reuse the execution mechanics while
> retaining separate protocol rules.
> * Migration remains phased, runnable, bounded, and diagnosable.
> h2. Scope and POC
> The initial migration is internal. It preserves Kafka protocols, public
> {{Consumer}} and {{ShareConsumer}} APIs, callback thread guarantees, runtime
> thread names, and the existing application/background thread topology.
> POC branch:
> [codex/async-consumer-reactor-poc|https://github.com/unknowntpo/kafka/tree/codex/async-consumer-reactor-poc]
> Implementation phase map:
> [consumer-reactor-poc-phase-map.md|https://github.com/unknowntpo/kafka/blob/cb7e5c4cb7619084d2e91f7efa494b35b1539d09/docs/design/consumer-reactor-poc-phase-map.md]
> The Confluence KIP is the authoritative design. POC class placement and
> compatibility adapters are implementation evidence, not part of the community
> API commitment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
