[
https://issues.apache.org/jira/browse/KAFKA-21506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reassigned KAFKA-21506:
---------------------------------------
Assignee: Matthias J. Sax
> Kafka Streams may report stales changelog endOffsets during "active-restore"
> phase
> ----------------------------------------------------------------------------------
>
> Key: KAFKA-21506
> URL: https://issues.apache.org/jira/browse/KAFKA-21506
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.4.0
> Reporter: Matthias J. Sax
> Assignee: Matthias J. Sax
> Priority: Major
> Fix For: 4.5.0, 4.4.1
>
>
> With the new "streams" rebalance protocol, Kafka Streams reports task-offset
> and task-end-offset information to the broker for standby/warmup tasks.
> However, if an active task needs to get restored, the state-updater-thread
> pauses to update standby/warmup tasks, as long as it's restoring the active
> task (this could take a longer period of time). During this phase, both
> task-offset and task-end-offset freeze. While this is correct for
> task-offset, it's incorrect for task-end-offset because the active task on
> some other member would still append new data to the changelog. (The freeze
> happens, because we currently update the task-end-offset information only
> based on the fetched record batches, which is "for free", ie does not require
> any additional RPC). As a side effect, the broker only sees incorrect lag
> information for these standby/warmup task over time (the observed lag is
> frozen while the real lag grows). If the last report told the broker that the
> standby/warmup is hot (ie, lag below acceptable.recovery.lag), the broker
> might incorrectly promote the standby/warmup to active in a rebalance or
> refiner stop even if the task did cool down in the mean time.
> To fix this issue, we should send explicit "get endOffset" request to the
> broker to refresh the task-end-offset correctly, for the pause partition as
> long as the state-updater is in "active restoring" phase (when we transit
> back to "standby updating" mode, we don't need the additional RPCs).
> The streams heartbeat thread sends a task-(end)-offset update only every
> 60sec by default (task offset reporting interval config), and thus we only
> need to send additional RPCs rarely enough that it's not a performance
> concern. We need to avoid blocking the state-updater-thread and should use a
> short RPC timeout – failures can be ignored (and can lead to reporting
> "unknown"), because we would retry after 60seconds again anyway.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)