ShivsundarR opened a new pull request, #20855: URL: https://github.com/apache/kafka/pull/20855
*What* - After KIP-1206, when `record_limit` mode was introduced, we ideally want no more than the #records in the `maxRecords` field in `ShareFetchRequest`. - Currently, the client broadcasts the share fetch requests to all nodes which host the leaders of the partitions that it is subscribed to. - The application thread would be woken up after the first response arrives, but meanwhile the responses from other nodes could bring in those many #records next and would wait in the buffer, that would mean we are wasting the acquisition locks for these records which are waiting. - Instead we would want to only send the next request when we poll again. - PR aims to send the request to only 1 node at a time in record_limit mode. - We are using partition-rotation on each poll so that no partition is starved. There were NCSS checkstyle errors in `ShareConsumeRequestManagerTest`, so added a few refactors there to reduce the length. Performance - When we have more consumers than the #partitions(i.e when real sharing of data happens in a partition), then we are seeing the performance is almost the same as the current approach. But when we have lesser consumers than the #partitions, then we see a performance regression as client is waiting for a node to return a response before it can send the next request. - Hence we have introduced this only for `record_limit` mode for now, future work will be done to improve this area. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
