[
https://issues.apache.org/jira/browse/KAFKA-21300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-21300:
------------------------------------
Component/s: group-coordinator
streams
> Streams group task ownership index is not rolled back, blocking valid task
> assignments
> --------------------------------------------------------------------------------------
>
> Key: KAFKA-21300
> URL: https://issues.apache.org/jira/browse/KAFKA-21300
> Project: Kafka
> Issue Type: Bug
> Components: group-coordinator, streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: group-coordinator, streams
>
> When the Streams group coordinator rolls back a task assignment change,
> member assignments and the process ownership index for standby and warm-up
> tasks can become inconsistent.
> In addition to per-member assignments, the coordinator maintains a set of
> process IDs that own each task. This index prevents the same task from being
> assigned to different members within the same process in conflicting roles.
> Although the ownership index is stored in a {{{}TimelineHashMap{}}}, its
> values are ordinary {{HashSet}} instances that are mutated in place.
> Reverting to a snapshot therefore does not restore the previous contents of
> these sets. Member assignments are restored correctly, but the index may
> retain ownership from a rolled-back assignment or lose ownership that should
> have been restored.
> A representative scenario involves a log append failure while adding a
> standby owner to a task that already has one:
> # P1 owns active task T, P3 owns standby task T, and P2 has no assigned
> tasks.
> # Increasing the number of standby replicas produces a target assignment
> that also assigns standby T to P2.
> # While processing P2’s heartbeat, the coordinator replays the
> current-assignment record in memory, changing the standby ownership index to
> {{{}{P3, P2}{}}}.
> # The log append fails and the coordinator rolls back. P2’s assignment
> becomes empty again, but P2 remains in the ownership index.
> # If P1 and P3 subsequently leave and P2 needs to receive active T, the
> coordinator blocks the assignment because it believes another member in
> process P2 still owns standby T.
> No actual owner exists to release the task. The member can continue sending
> successful heartbeats while retaining an empty assignment and remaining in
> {{{}UNRELEASED_TASKS{}}}. For an application with a single task, this can
> halt input processing.
> Removing the original member and rejoining with the same process ID may not
> resolve the issue. Since the original member’s assignment is empty after
> rollback, member removal does not clean up the stale ownership entry.
> Restarting an application with persistent stores while preserving its state
> directory can reuse the same process ID and meet this condition.
> The reverse inconsistency can also occur. If a standby or warm-up owner is
> removed and that change is rolled back, the member’s assignment is restored,
> but its process ID may remain absent from the ownership index. The check that
> should prevent another member in the same process from receiving the active
> task can then incorrectly allow the assignment.
> The impact can persist after such a rollback while the same coordinator state
> remains loaded. Reconstructing coordinator state from persisted member
> records also rebuilds the ownership index.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
