frankvicky commented on PR #16833: URL: https://github.com/apache/kafka/pull/16833#issuecomment-2301334628
Hi @chia7712 and @lianetm, Yes, you are right, we could remove the `failAndRemoveExpiredCommitRequests`, but we also need to consider the side effects. I’ve identified a side effect of removing both `failAndRemoveXXX` methods, and I’ll use `failAndRemoveExpiredCommitRequests` as an example. I’ve reviewed the code, and I believe we need to consider the impact of exponential backoff once we remove `failAndRemoveExpiredCommitRequests`. Before its removal, expired requests(the request exceed `deadlineMs`) were removed immediately by the `failAndRemoveExpiredCommitRequests` method. https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L853-L859 https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/clients/src/main/java/org/apache/kafka/common/utils/Timer.java#L70-L72 After its removal, the request will only be cleared once the callback is triggered: https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L452-L459 This means that after `networkClientDelegate.poll()`, the request will remain until it reaches the Network Layer, which might take a long time due to exponential backoff: https://github.com/apache/kafka/blob/944c1353a925858ea9bd9024a713cd7301f55133/clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java#L169-L187 I’m not sure about the side effects of removing both `failAndRemoveXXX` methods. Could this change increase the pressure on buffers (`unsentOffsetFetches` and `unsentOffsetCommits`)? The concern is that requests might take longer to be removed, potentially leading to a buildup in the buffer and causing memory management issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
