Matthias J. Sax created KAFKA-21576:
---------------------------------------

             Summary: Streams client reports a restored but not yet running 
active task without an end offset, so its task moves cold
                 Key: KAFKA-21576
                 URL: https://issues.apache.org/jira/browse/KAFKA-21576
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 4.4.0
            Reporter: Matthias J. Sax
            Assignee: Matthias J. Sax
             Fix For: 4.5.0, 4.4.1


Under the streams group protocol, a client reports the changelog offsets of the 
active tasks it is restoring. The coordinator treats an active task with a 
reported offset as restoring, and with an end offset, it can tell whether the 
task is caught up.

An active task that has finished restoring but is not running yet is reported 
with an offset but without an end offset:
 * TaskManager.maybeUpdateTaskOffsetSumSnapshot() reports the offset of every 
task that is not RUNNING.
 * DefaultStateUpdater.updateTaskEndOffsetSumSnapshot() computes end offsets 
only for the tasks the state updater is still restoring. A restored task 
waiting in restoredActiveTasks is not among them.

The coordinator therefore sees the owner as restoring with an unknown lag. The 
warm-up refiner then counts the owner's state as not usable, and moves a task 
the target assignment gives to another member without warming it up first.

The window lasts as long as the group is not ready: StreamThread.runOnce() 
returns before draining restored tasks while streamsGroupReady is false. A 
member whose first tasks arrive together with ASSIGNMENT_DELAYED (for example, 
because another member joined within group.streams.assignment.interval.ms) 
stays not ready until the next assignment, and reports its restored active 
tasks with offsets but without end offsets until then. If the coordinator 
refines the assignment in that time, it moves the member's tasks without 
warming them up.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to