Jason Gustafson created KAFKA-4007:
--------------------------------------

             Summary: Improve fetch pipelining for low values of 
max.poll.records
                 Key: KAFKA-4007
                 URL: https://issues.apache.org/jira/browse/KAFKA-4007
             Project: Kafka
          Issue Type: Improvement
          Components: consumer
            Reporter: Jason Gustafson


Currently the consumer will only send a prefetch for a partition after all the 
records from the previous fetch have been consumed. This can lead to suboptimal 
pipelining when max.poll.records is set very low since the processing latency 
for a small set of records may be small compared to the latency of a fetch. An 
improvement suggested by [~junrao] is to send the fetch anyway even if we have 
unprocessed data buffered, but delay reading it from the socket until that data 
has been consumed. Potentially the consumer can delay reading _any_ pending 
fetch until it is ready to be returned to the user, which may help control 
memory better. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to