sanghyeok An created KAFKA-21447:
------------------------------------
Summary: Streams consumer can make newly assigned partitions
fetchable by applying a pending assignment after fencing
Key: KAFKA-21447
URL: https://issues.apache.org/jira/browse/KAFKA-21447
Project: Kafka
Issue Type: Bug
Reporter: sanghyeok An
Assignee: sanghyeok An
When using the Streams rebalance protocol, a member may be fenced while an
assignment received in a successful {{StreamsGroupHeartbeat}} response is still
waiting to be applied. The pending assignment can then be applied after fencing.
{{StreamsMembershipManager.onFenced()}} marks only the partitions currently
present in the subscription as pending revocation. However, a subsequently
processed assignment application event updates the subscription without
checking whether the member is already in the {{FENCED}} state.
As a result, partitions added after fencing do not have {{pendingRevocation}}
set. Once assignment callback completion is processed and Streams completes
task initialization and offset initialization, these partitions can become
fetchable while the member is still {{{}FENCED{}}}.
This does *not* require the broker to send a new assignment after fencing. It
involves the client applying an assignment that it received successfully
{*}before fencing{*}.
||Step||Network thread||Application thread||Applied assignment and B’s state||
|1|Completes assignment of A.|Processes A.|A|
|2|Receives A+B in a successful heartbeat response and queues a
{{{}StreamsTasksAssignedEvent{}}}.|Has not processed the event yet.|A|
|3|—|Drains the background event queue, obtaining the A+B event. No lost event
is present at this point.|A|
|4|Receives {{FENCED_MEMBER_EPOCH}} or {{UNKNOWN_MEMBER_ID}} in a subsequent
heartbeat response. {{onFenced()}} marks A as pending revocation and queues a
lost event.|Processes the previously drained A+B event.|A; fetching from A is
blocked.|
|5|—|Sends an {{ApplyAssignmentEvent}} for A+B and waits for the result.|A|
|6|Applies A+B without checking the member state. A retains its
pending-revocation flag, but B does not have that flag.|Waits for assignment
application to complete.|A+B; B is awaiting the assignment callback.|
|7|—|Creates B’s task in the assignment callback, pauses B, and sends a
callback completion event.|A+B; B is paused.|
|8|Processes callback completion and clears B’s
{{{}pendingOnAssignedCallback{}}}.|The current poll returns, for example
because its timeout expires. The lost event remains pending until the next
background event processing pass.|A+B; B remains paused.|
|9|—|After poll returns, completes B’s task initialization through the
StateUpdater and calls {{{}mainConsumer.resume(B){}}}.|A+B; B is resumed, but
may still have no position.|
|10|Processes the resume event.|In the next poll, first submits an
{{{}AsyncPollEvent{}}}, then processes the pending lost callback.|A+B|
|11|Initializes B’s position while processing the {{{}AsyncPollEvent{}}}.
OffsetFetch returns no committed offset, so ListOffsets is used to retrieve the
earliest offset.|Executes the lost callback, or has sent its completion
notification but the network thread has not processed it yet.|A+B|
|12|Processes the offset responses while B remains assigned and sets a valid
position. B becomes fetchable.|Continues lost handling.|A+B; B is fetchable.|
|13|If B’s fetch response is processed before lost completion, its records can
enter the fetch buffer.|After returning from the lost callback, may return B’s
records before assignment removal is reflected in the subscription.|A+B|
|14|Processes the lost completion event, clears the entire assignment, and
proceeds to rejoin.|Continues processing.|Empty assignment|
The background events drained in step 3 are a snapshot of the queue at that
time. A lost event added afterward is not necessarily included in the same
processing pass. Therefore, the poll handling the assignment event can return,
allowing task initialization and resume to occur before the next lost callback
is processed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)