sanghyeok An created KAFKA-21512:
------------------------------------
Summary: Streams member is fenced when retrying a heartbeat after
losing a standby promotion response
Key: KAFKA-21512
URL: https://issues.apache.org/jira/browse/KAFKA-21512
Project: Kafka
Issue Type: Bug
Reporter: sanghyeok An
Assignee: sanghyeok An
In a Streams group, if a heartbeat response promoting a member’s standby task
to active is lost, the client’s normal retry is rejected with
{{{}FENCED_MEMBER_EPOCH{}}}. The member consequently releases its entire
assignment, including unrelated active tasks it was still processing, and
rejoins the group.
The coordinator tolerates a lost response containing an epoch bump by accepting
a request at the previous member epoch when the reported assignment is a subset
of the current assignment. However, Streams checks active, standby, and warmup
tasks separately against their corresponding roles. This rejects a legitimate
retry when a task has been promoted from standby to active on the same member.
The issue can occur in the following sequence:
# A group using the Streams protocol contains members A and M with different
process IDs. It uses the built-in sticky assignor, the default assignment
refiner, and {{streams.num.standby.replicas=1}} for stateful tasks T and U. The
group has the following stable assignment, and M’s member epoch is {{{}10{}}}.
||Member||Active||Standby||
|A|T|U|
|M|U|T|
# A leaves gracefully, and its removal is applied on the coordinator. After
the assignment interval has elapsed, M sends a heartbeat with epoch {{{}10{}}}.
# The coordinator assigns both T and U as active tasks to the remaining member
M. Since T’s previous active owner has already been removed and no other tasks
require revocation, the coordinator promotes T from standby to active and
advances M’s epoch to {{11}} within the same heartbeat. These changes are
persisted.
# The successful heartbeat response containing the promotion is lost, for
example because the connection closes before the client receives it. M is
unaware of the new epoch and assignment. After reconnecting, and before its
session or max-poll timeout expires, it retries with epoch {{{}10{}}}, active
tasks {{{}{U}{}}}, and standby tasks {{{}{T}{}}}.
# Although the previous epoch matches and U remains assigned to M as active,
the coordinator returns {{FENCED_MEMBER_EPOCH}} because T is no longer in M’s
current standby assignment.
At the time of the retry, the states are:
||State||Member epoch||Active||Standby||
|Client request|10|U|T|
|Coordinator|11, previous=10|T, U|None|
T has not moved to another member; {*}it has changed from standby to active on
the same member M{*}. However,
{{GroupMetadataManager.throwIfStreamsGroupMemberEpochIsInvalid()}} compares the
reported standby task {{{T}}} only against the current standby assignment,
treating the request as stale ownership. The request is rejected before
reaching the logic that resends the current assignment.
The expected behavior is to return the current epoch and promoted assignment
for this legitimate retry, allowing the member to converge. Instead, the client
handles fencing by stopping fetches for all assigned partitions, including
those belonging to U, invoking {{{}onAllTasksLost{}}}, clearing its local
assignment, and rejoining the group.
Automatic recovery is possible once connectivity is restored and the callback
completes. The direct impact is therefore {*}a processing interruption caused
by unnecessary release of the entire assignment and rejoining{*}, affecting
both the promoted task T and the otherwise unaffected active task U.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)