[
https://issues.apache.org/jira/browse/KAFKA-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sanghyeok An updated KAFKA-21512:
---------------------------------
Component/s: group-coordinator
streams
> Streams member is fenced when retrying a heartbeat after losing a standby
> promotion response
> --------------------------------------------------------------------------------------------
>
> Key: KAFKA-21512
> URL: https://issues.apache.org/jira/browse/KAFKA-21512
> Project: Kafka
> Issue Type: Bug
> Components: group-coordinator, streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
>
> In a Streams group, if a heartbeat response promoting a member’s standby task
> to active is lost, the client’s normal retry is rejected with
> {{{}FENCED_MEMBER_EPOCH{}}}. The member consequently releases its entire
> assignment, including unrelated active tasks it was still processing, and
> rejoins the group.
> The coordinator tolerates a lost response containing an epoch bump by
> accepting a request at the previous member epoch when the reported assignment
> is a subset of the current assignment. However, Streams checks active,
> standby, and warmup tasks separately against their corresponding roles. This
> rejects a legitimate retry when a task has been promoted from standby to
> active on the same member.
> The issue can occur in the following sequence:
> 1. A group using the Streams protocol contains members A and M with different
> process IDs. It uses the built-in sticky assignor, the default assignment
> refiner, and {{streams.num.standby.replicas=1}} for stateful tasks T and U.
> The group has the following stable assignment, and M’s member epoch is
> {{{}10{}}}.
> ||Member||Active||Standby||
> |A|T|U|
> |M|U|T|
> 2. A leaves gracefully, and its removal is applied on the coordinator. After
> the assignment interval has elapsed, M sends a heartbeat with epoch
> {{{}10{}}}.
> 3. The coordinator assigns both T and U as active tasks to the remaining
> member M. Since T’s previous active owner has already been removed and no
> other tasks require revocation, the coordinator promotes T from standby to
> active and advances M’s epoch to {{11}} within the same heartbeat. These
> changes are persisted.
> 4. The successful heartbeat response containing the promotion is lost, for
> example because the connection closes before the client receives it. M is
> unaware of the new epoch and assignment. After reconnecting, and before its
> session or max-poll timeout expires, it retries with epoch {{{}10{}}}, active
> tasks \{U}, and standby tasks \{T}.
> 5. Although the previous epoch matches and U remains assigned to M as active,
> the coordinator returns {{FENCED_MEMBER_EPOCH}} because T is no longer in M’s
> current standby assignment.
> At the time of the retry, the states are:
> ||State||Member epoch||Active||Standby||
> |Client request|10|U|T|
> |Coordinator|11, previous=10|T, U|None|
> T has not moved to another member; {*}it has changed from standby to active
> on the same member M{*}. However,
> {{GroupMetadataManager.throwIfStreamsGroupMemberEpochIsInvalid()}} compares
> the reported standby task \{T} only against the current standby assignment,
> treating the request as stale ownership. The request is rejected before
> reaching the logic that resends the current assignment.
>
> The expected behavior is to return the current epoch and promoted assignment
> for this legitimate retry, allowing the member to converge. Instead, the
> client handles fencing by stopping fetches for all assigned partitions,
> including those belonging to U, invoking {{{}onAllTasksLost{}}}, clearing its
> local assignment, and rejoining the group.
>
> Automatic recovery is possible once connectivity is restored and the callback
> completes. The direct impact is therefore {*}a processing interruption caused
> by unnecessary release of the entire assignment and rejoining{*}, affecting
> both the promoted task T and the otherwise unaffected active task U.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)