[ 
https://issues.apache.org/jira/browse/KAFKA-21506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-21506:
---------------------------------------

    Assignee: Matthias J. Sax

> Kafka Streams may report stales changelog endOffsets during "active-restore" 
> phase
> ----------------------------------------------------------------------------------
>
>                 Key: KAFKA-21506
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21506
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.4.0
>            Reporter: Matthias J. Sax
>            Assignee: Matthias J. Sax
>            Priority: Major
>             Fix For: 4.5.0, 4.4.1
>
>
> With the new "streams" rebalance protocol, Kafka Streams reports task-offset 
> and task-end-offset information to the broker for standby/warmup tasks. 
> However, if an active task needs to get restored, the state-updater-thread 
> pauses to update standby/warmup tasks, as long as it's restoring the active 
> task (this could take a longer period of time). During this phase, both 
> task-offset and task-end-offset freeze. While this is correct for 
> task-offset, it's incorrect for task-end-offset because the active task on 
> some other member would still append new data to the changelog. (The freeze 
> happens, because we currently update the task-end-offset information only 
> based on the fetched record batches, which is "for free", ie does not require 
> any additional RPC). As a side effect, the broker only sees incorrect lag 
> information for these standby/warmup task over time (the observed lag is 
> frozen while the real lag grows). If the last report told the broker that the 
> standby/warmup is hot (ie, lag below acceptable.recovery.lag), the broker 
> might incorrectly promote the standby/warmup to active in a rebalance or 
> refiner stop even if the task did cool down in the mean time.
> To fix this issue, we should send explicit "get endOffset" request to the 
> broker to refresh the task-end-offset correctly, for the pause partition as 
> long as the state-updater is in "active restoring" phase (when we transit 
> back to "standby updating" mode, we don't need the additional RPCs).
> The streams heartbeat thread sends a task-(end)-offset update only every 
> 60sec by default (task offset reporting interval config), and thus we only 
> need to send additional RPCs rarely enough that it's not a performance 
> concern. We need to avoid blocking the state-updater-thread and should use a 
> short RPC timeout – failures can be ignored (and can lead to reporting 
> "unknown"), because we would retry after 60seconds again anyway.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to