This is a good observation on limiting total memory usage. If I understand the proposal I think it is that the consumer client would stop sending fetch requests once a certain number of in-flight fetch requests is met. I think a better approach would be to always issue one fetch request to each broker immediately, allow the server to process that request, and send data back to the local machine where it would be stored in the socket buffer (up to that buffer size). Instead of throttling the requests sent, the consumer should ideally throttle the responses read from the socket buffer at any given time. That is, in a single poll call, rather than reading from every single socket it should just read until it has a given amount of memory used then bail out early. It can come back and read more from the other sockets after those messages are processed.
The advantage of this approach is that you don't incur the additional latency. -Jay On Mon, Oct 10, 2016 at 6:41 AM, Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi all, > > I would like to discuss the following KIP proposal: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 81%3A+Max+in-flight+fetches > > > Feedback and comments are welcome. > Thanks ! > > Mickael >
