sanghyeok An created KAFKA-21255:
------------------------------------
Summary: Streams heartbeat retry responses can omit assignments
after response loss
Key: KAFKA-21255
URL: https://issues.apache.org/jira/browse/KAFKA-21255
Project: Kafka
Issue Type: Bug
Reporter: sanghyeok An
Assignee: sanghyeok An
When a Streams group coordinator applies an assignment change but the heartbeat
response is lost, the server and client can retain different assignments.
After a heartbeat failure, the client retries with its current active, standby,
and warmup tasks. However, the server has already applied the assignment change
while processing the previous request. If processing the retry produces no
further assignment change, the server omits the task fields from its response.
Consequently, the assignment that the client missed is not delivered again.
The following scenarios illustrate the issue. These examples use only active
tasks; the client reports empty lists ({{{}[]{}}}) for standby and warmup tasks
when retrying.
*Scenario 1: Losing an assignment response that advances the member epoch*
The client owns task {{[0]}} at member epoch {{{}1{}}}. While processing a
heartbeat, the server advances the member epoch to {{2}} and assigns tasks
{{{}[0, 1]{}}}, but the response is lost.
The client retries with its existing epoch {{1}} and owned task {{{}[0]{}}}.
Since the server has already recorded epoch {{2}} and assignment {{{}[0,
1]{}}}, the retry response returns epoch {{2}} but omits the task fields.
The client updates its epoch to {{2}} while continuing to own only task
{{{}[0]{}}}. The server considers task {{1}} assigned, but the client does not
process it. This discrepancy can persist even while subsequent heartbeats
succeed.
*Scenario 2: Losing a revocation response within the same member epoch*
The client owns tasks {{[0, 1]}} at member epoch {{{}1{}}}, and its new target
assignment is {{{}[0]{}}}. The server transitions the member to
{{{}UNREVOKED_TASKS{}}}, waiting for task {{1}} to be released, and responds
with epoch {{1}} and the remaining task {{{}[0]{}}}. This response is lost.
Because the client did not receive the revocation instruction, it retries with
epoch {{1}} and owned tasks {{{}[0, 1]{}}}. The server remains in the same
state and omits the task fields because its assignment has not changed.
The client continues to own task {{1}} and cannot acknowledge its revocation.
Moving that task to another member is delayed. When the existing rebalance
timeout expires, the member is removed and must rejoin.
*Scenario 3: Losing an additional assignment within the same member epoch*
The client has already reached member epoch {{{}2{}}}, but the member is in
{{{}UNRELEASED_TASKS{}}}, waiting for another member to release a task. Once
the previous owner releases it, the server assigns the task without changing
member epoch {{{}2{}}}. This response is lost.
The client retries with epoch {{2}} and the tasks it owned before the
additional assignment. Since the server has already recorded the assignment, it
omits the task fields from the retry response.
The server and client have the same member epoch but different assignments. The
client does not process the task that the server considers assigned, and
successful heartbeats alone may not resolve the discrepancy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
