sanghyeok An created KAFKA-21447:
------------------------------------

             Summary: Streams consumer can make newly assigned partitions 
fetchable by applying a pending assignment after fencing
                 Key: KAFKA-21447
                 URL: https://issues.apache.org/jira/browse/KAFKA-21447
             Project: Kafka
          Issue Type: Bug
            Reporter: sanghyeok An
            Assignee: sanghyeok An


When using the Streams rebalance protocol, a member may be fenced while an 
assignment received in a successful {{StreamsGroupHeartbeat}} response is still 
waiting to be applied. The pending assignment can then be applied after fencing.

{{StreamsMembershipManager.onFenced()}} marks only the partitions currently 
present in the subscription as pending revocation. However, a subsequently 
processed assignment application event updates the subscription without 
checking whether the member is already in the {{FENCED}} state.

As a result, partitions added after fencing do not have {{pendingRevocation}} 
set. Once assignment callback completion is processed and Streams completes 
task initialization and offset initialization, these partitions can become 
fetchable while the member is still {{{}FENCED{}}}.

This does *not* require the broker to send a new assignment after fencing. It 
involves the client applying an assignment that it received successfully 
{*}before fencing{*}.

 
||Step||Network thread||Application thread||Applied assignment and B’s state||
|1|Completes assignment of A.|Processes A.|A|
|2|Receives A+B in a successful heartbeat response and queues a 
{{{}StreamsTasksAssignedEvent{}}}.|Has not processed the event yet.|A|
|3|—|Drains the background event queue, obtaining the A+B event. No lost event 
is present at this point.|A|
|4|Receives {{FENCED_MEMBER_EPOCH}} or {{UNKNOWN_MEMBER_ID}} in a subsequent 
heartbeat response. {{onFenced()}} marks A as pending revocation and queues a 
lost event.|Processes the previously drained A+B event.|A; fetching from A is 
blocked.|
|5|—|Sends an {{ApplyAssignmentEvent}} for A+B and waits for the result.|A|
|6|Applies A+B without checking the member state. A retains its 
pending-revocation flag, but B does not have that flag.|Waits for assignment 
application to complete.|A+B; B is awaiting the assignment callback.|
|7|—|Creates B’s task in the assignment callback, pauses B, and sends a 
callback completion event.|A+B; B is paused.|
|8|Processes callback completion and clears B’s 
{{{}pendingOnAssignedCallback{}}}.|The current poll returns, for example 
because its timeout expires. The lost event remains pending until the next 
background event processing pass.|A+B; B remains paused.|
|9|—|After poll returns, completes B’s task initialization through the 
StateUpdater and calls {{{}mainConsumer.resume(B){}}}.|A+B; B is resumed, but 
may still have no position.|
|10|Processes the resume event.|In the next poll, first submits an 
{{{}AsyncPollEvent{}}}, then processes the pending lost callback.|A+B|
|11|Initializes B’s position while processing the {{{}AsyncPollEvent{}}}. 
OffsetFetch returns no committed offset, so ListOffsets is used to retrieve the 
earliest offset.|Executes the lost callback, or has sent its completion 
notification but the network thread has not processed it yet.|A+B|
|12|Processes the offset responses while B remains assigned and sets a valid 
position. B becomes fetchable.|Continues lost handling.|A+B; B is fetchable.|
|13|If B’s fetch response is processed before lost completion, its records can 
enter the fetch buffer.|After returning from the lost callback, may return B’s 
records before assignment removal is reflected in the subscription.|A+B|
|14|Processes the lost completion event, clears the entire assignment, and 
proceeds to rejoin.|Continues processing.|Empty assignment|

The background events drained in step 3 are a snapshot of the queue at that 
time. A lost event added afterward is not necessarily included in the same 
processing pass. Therefore, the poll handling the assignment event can return, 
allowing task initialization and resume to occur before the next lost callback 
is processed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to