Matthias J. Sax created KAFKA-21506:
---------------------------------------
Summary: Kafka Streams may report stales changelog endOffsets
during "active-restore" phase
Key: KAFKA-21506
URL: https://issues.apache.org/jira/browse/KAFKA-21506
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.4.0
Reporter: Matthias J. Sax
Fix For: 4.5.0, 4.4.1
With the new "streams" rebalance protocol, Kafka Streams reports task-offset
and task-end-offset information to the broker for standby/warmup tasks.
However, if an active task needs to get restored, the state-updater-thread
pauses to update standby/warmup tasks, as long as it's restoring the active
task (this could take a longer period of time). During this phase, both
task-offset and task-end-offset freeze. While this is correct for task-offset,
it's incorrect for task-end-offset because the active task on some other member
would still append new data to the changelog. (The freeze happens, because we
currently update the task-end-offset information only based on the fetched
record batches, which is "for free", ie does not require any additional RPC).
As a side effect, the broker only sees incorrect lag information for these
standby/warmup task over time (the observed lag is frozen while the real lag
grows). If the last report told the broker that the standby/warmup is hot (ie,
lag below acceptable.recovery.lag), the broker might incorrectly promote the
standby/warmup to active in a rebalance or refiner stop even if the task did
cool down in the mean time.
To fix this issue, we should send explicit "get endOffset" request to the
broker to refresh the task-end-offset correctly, for the pause partition as
long as the state-updater is in "active restoring" phase (when we transit back
to "standby updating" mode, we don't need the additional RPCs).
The streams heartbeat thread sends a task-(end)-offset update only every 60sec
by default (task offset reporting interval config), and thus we only need to
send additional RPCs rarely enough that it's not a performance concern.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)