aloknnikhil opened a new pull request #11204: URL: https://github.com/apache/kafka/pull/11204
**Forward porting** - https://github.com/linkedin/kafka/pull/186 **TICKET** = [KAFKA-13188](https://issues.apache.org/jira/browse/KAFKA-13188) **LI_DESCRIPTION** = The current KafkaConsumer code includes support for allocating buffers from MemoryPool when allocating bytes for requests to Brokers. However, the code doesn't release them back to the pool and hence, rendering pooling moot. Currently, it works because it uses MemoryPool.NONE which just mallocs buffer every time a buffer is requested. However, this won't work if a different memory pool implementation is used. The consumer can't just release the pool back into the memory because the fetch requests keep on holding on to the references to the pool in CompletedFetch objects which live across multiple poll calls. The idea here is to use ref counting based approach, where the ClientResponse increments ref count every time a CompletedFetch object is created and decrement when the fetch is drained after a poll calls returning records. For the rest of the things such as metadata, offset commit, list offsets it is somewhat easier as the client is done with the response bytes after response future callback is completed. This PR also adds the ClientResponseWithFinalize class for debugging purposes as well protected by linkedin.enable.client.resonse.leakcheck flag. The class uses finalizer to check whether there is some issue in code due to which the buffer wasn't released back to the pool yet. However, during in an actual production setting, the finalizers are costly and hence, not enabled by default. **EXIT_CRITERIA** = If and when the upstream ticket is merged, and the changes are pulled in ### Committer Checklist (excluded from commit message) - [x] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
