sanghyeok An created KAFKA-21300:
------------------------------------
Summary: Streams group task ownership index is not rolled back,
blocking valid task assignments
Key: KAFKA-21300
URL: https://issues.apache.org/jira/browse/KAFKA-21300
Project: Kafka
Issue Type: Bug
Reporter: sanghyeok An
Assignee: sanghyeok An
When the Streams group coordinator rolls back a task assignment change, member
assignments and the process ownership index for standby and warm-up tasks can
become inconsistent.
In addition to per-member assignments, the coordinator maintains a set of
process IDs that own each task. This index prevents the same task from being
assigned to different members within the same process in conflicting roles.
Although the ownership index is stored in a {{{}TimelineHashMap{}}}, its values
are ordinary {{HashSet}} instances that are mutated in place. Reverting to a
snapshot therefore does not restore the previous contents of these sets. Member
assignments are restored correctly, but the index may retain ownership from a
rolled-back assignment or lose ownership that should have been restored.
A representative scenario involves a log append failure while adding a standby
owner to a task that already has one:
# P1 owns active task T, P3 owns standby task T, and P2 has no assigned tasks.
# Increasing the number of standby replicas produces a target assignment that
also assigns standby T to P2.
# While processing P2’s heartbeat, the coordinator replays the
current-assignment record in memory, changing the standby ownership index to
{{{}{P3, P2}{}}}.
# The log append fails and the coordinator rolls back. P2’s assignment becomes
empty again, but P2 remains in the ownership index.
# If P1 and P3 subsequently leave and P2 needs to receive active T, the
coordinator blocks the assignment because it believes another member in process
P2 still owns standby T.
No actual owner exists to release the task. The member can continue sending
successful heartbeats while retaining an empty assignment and remaining in
{{{}UNRELEASED_TASKS{}}}. For an application with a single task, this can halt
input processing.
Removing the original member and rejoining with the same process ID may not
resolve the issue. Since the original member’s assignment is empty after
rollback, member removal does not clean up the stale ownership entry.
Restarting an application with persistent stores while preserving its state
directory can reuse the same process ID and meet this condition.
The reverse inconsistency can also occur. If a standby or warm-up owner is
removed and that change is rolled back, the member’s assignment is restored,
but its process ID may remain absent from the ownership index. The check that
should prevent another member in the same process from receiving the active
task can then incorrectly allow the assignment.
The impact can persist after such a rollback while the same coordinator state
remains loaded. Reconstructing coordinator state from persisted member records
also rebuilds the ownership index.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
