sanghyeok An created KAFKA-21512:
------------------------------------

             Summary: Streams member is fenced when retrying a heartbeat after 
losing a standby promotion response
                 Key: KAFKA-21512
                 URL: https://issues.apache.org/jira/browse/KAFKA-21512
             Project: Kafka
          Issue Type: Bug
            Reporter: sanghyeok An
            Assignee: sanghyeok An


In a Streams group, if a heartbeat response promoting a member’s standby task 
to active is lost, the client’s normal retry is rejected with 
{{{}FENCED_MEMBER_EPOCH{}}}. The member consequently releases its entire 
assignment, including unrelated active tasks it was still processing, and 
rejoins the group.

The coordinator tolerates a lost response containing an epoch bump by accepting 
a request at the previous member epoch when the reported assignment is a subset 
of the current assignment. However, Streams checks active, standby, and warmup 
tasks separately against their corresponding roles. This rejects a legitimate 
retry when a task has been promoted from standby to active on the same member.

The issue can occur in the following sequence:
 # A group using the Streams protocol contains members A and M with different 
process IDs. It uses the built-in sticky assignor, the default assignment 
refiner, and {{streams.num.standby.replicas=1}} for stateful tasks T and U. The 
group has the following stable assignment, and M’s member epoch is {{{}10{}}}.
||Member||Active||Standby||
|A|T|U|
|M|U|T|

 # A leaves gracefully, and its removal is applied on the coordinator. After 
the assignment interval has elapsed, M sends a heartbeat with epoch {{{}10{}}}.

 # The coordinator assigns both T and U as active tasks to the remaining member 
M. Since T’s previous active owner has already been removed and no other tasks 
require revocation, the coordinator promotes T from standby to active and 
advances M’s epoch to {{11}} within the same heartbeat. These changes are 
persisted.

 # The successful heartbeat response containing the promotion is lost, for 
example because the connection closes before the client receives it. M is 
unaware of the new epoch and assignment. After reconnecting, and before its 
session or max-poll timeout expires, it retries with epoch {{{}10{}}}, active 
tasks {{{}{U}{}}}, and standby tasks {{{}{T}{}}}.

 # Although the previous epoch matches and U remains assigned to M as active, 
the coordinator returns {{FENCED_MEMBER_EPOCH}} because T is no longer in M’s 
current standby assignment.

 

At the time of the retry, the states are:
||State||Member epoch||Active||Standby||
|Client request|10|U|T|
|Coordinator|11, previous=10|T, U|None|

T has not moved to another member; {*}it has changed from standby to active on 
the same member M{*}. However, 
{{GroupMetadataManager.throwIfStreamsGroupMemberEpochIsInvalid()}} compares the 
reported standby task {{{T}}} only against the current standby assignment, 
treating the request as stale ownership. The request is rejected before 
reaching the logic that resends the current assignment.

 

The expected behavior is to return the current epoch and promoted assignment 
for this legitimate retry, allowing the member to converge. Instead, the client 
handles fencing by stopping fetches for all assigned partitions, including 
those belonging to U, invoking {{{}onAllTasksLost{}}}, clearing its local 
assignment, and rejoining the group.

 

Automatic recovery is possible once connectivity is restored and the callback 
completes. The direct impact is therefore {*}a processing interruption caused 
by unnecessary release of the entire assignment and rejoining{*}, affecting 
both the promoted task T and the otherwise unaffected active task U.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to