lianetm commented on PR #14690: URL: https://github.com/apache/kafka/pull/14690#issuecomment-1794951522
Comment regarding the async reconciliation process and callback execution. I get your concerns @philipnee , but this PR does not include callback execution for now, the membership manager triggers the callbacks and needs to know when they complete. I expect that will follow in a separate PR based on events as @AndrewJSchofield mentioned. Without having got into the implementation details yet, I expect that the membershipManager (background thread) will enqueue an app event to execute the callbacks in the app thread, and when that app thread completes the callbacks, it will enqueue a background event to notify the manager about the completion (btw, nice here again that according to the protocol there will be only one assignment being reconciled at a time @dajac). The async nature of the membership manager in this PR it is not only due to the callbacks. The reconciliation process handles 3 main async operations: metadata, commit, callbacks (triggering and notification when completed, no execution). By basing the them on completable futures we ensure that the background thread continues it operations while there is a reconciliation in process. Note on the reconciliation not being time bounded on the client. The reasoning is that the time boundaries are set by the broker, that keeps a rebalance timer. If the reconciliation takes longer than allowed (ex. stuck in any of the 3 async operations), the expectation is that the broker will re-assign the partitions and kick the member out of the group. So from the client side we just care about triggering the async reconciliation, making sure we keep sending HB and processing responses (will let us know if kicked-out), and make sure that when reconciliation completes we check if it is still relevant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
