[ 
https://issues.apache.org/jira/browse/KAFKA-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sanghyeok An updated KAFKA-21512:
---------------------------------
    Component/s: group-coordinator
                 streams

> Streams member is fenced when retrying a heartbeat after losing a standby 
> promotion response
> --------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-21512
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21512
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator, streams
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Minor
>
> In a Streams group, if a heartbeat response promoting a member’s standby task 
> to active is lost, the client’s normal retry is rejected with 
> {{{}FENCED_MEMBER_EPOCH{}}}. The member consequently releases its entire 
> assignment, including unrelated active tasks it was still processing, and 
> rejoins the group.
> The coordinator tolerates a lost response containing an epoch bump by 
> accepting a request at the previous member epoch when the reported assignment 
> is a subset of the current assignment. However, Streams checks active, 
> standby, and warmup tasks separately against their corresponding roles. This 
> rejects a legitimate retry when a task has been promoted from standby to 
> active on the same member.
> The issue can occur in the following sequence:
> 1. A group using the Streams protocol contains members A and M with different 
> process IDs. It uses the built-in sticky assignor, the default assignment 
> refiner, and {{streams.num.standby.replicas=1}} for stateful tasks T and U. 
> The group has the following stable assignment, and M’s member epoch is 
> {{{}10{}}}.
> ||Member||Active||Standby||
> |A|T|U|
> |M|U|T|
> 2. A leaves gracefully, and its removal is applied on the coordinator. After 
> the assignment interval has elapsed, M sends a heartbeat with epoch 
> {{{}10{}}}.
> 3. The coordinator assigns both T and U as active tasks to the remaining 
> member M. Since T’s previous active owner has already been removed and no 
> other tasks require revocation, the coordinator promotes T from standby to 
> active and advances M’s epoch to {{11}} within the same heartbeat. These 
> changes are persisted.
> 4. The successful heartbeat response containing the promotion is lost, for 
> example because the connection closes before the client receives it. M is 
> unaware of the new epoch and assignment. After reconnecting, and before its 
> session or max-poll timeout expires, it retries with epoch {{{}10{}}}, active 
> tasks \{U}, and standby tasks \{T}.
> 5. Although the previous epoch matches and U remains assigned to M as active, 
> the coordinator returns {{FENCED_MEMBER_EPOCH}} because T is no longer in M’s 
> current standby assignment.
> At the time of the retry, the states are:
> ||State||Member epoch||Active||Standby||
> |Client request|10|U|T|
> |Coordinator|11, previous=10|T, U|None|
> T has not moved to another member; {*}it has changed from standby to active 
> on the same member M{*}. However, 
> {{GroupMetadataManager.throwIfStreamsGroupMemberEpochIsInvalid()}} compares 
> the reported standby task \{T} only against the current standby assignment, 
> treating the request as stale ownership. The request is rejected before 
> reaching the logic that resends the current assignment.
>  
> The expected behavior is to return the current epoch and promoted assignment 
> for this legitimate retry, allowing the member to converge. Instead, the 
> client handles fencing by stopping fetches for all assigned partitions, 
> including those belonging to U, invoking {{{}onAllTasksLost{}}}, clearing its 
> local assignment, and rejoining the group.
>  
> Automatic recovery is possible once connectivity is restored and the callback 
> completes. The direct impact is therefore {*}a processing interruption caused 
> by unnecessary release of the entire assignment and rejoining{*}, affecting 
> both the promoted task T and the otherwise unaffected active task U.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to