Jason Gustafson created KAFKA-4007: -------------------------------------- Summary: Improve fetch pipelining for low values of max.poll.records Key: KAFKA-4007 URL: https://issues.apache.org/jira/browse/KAFKA-4007 Project: Kafka Issue Type: Improvement Components: consumer Reporter: Jason Gustafson
Currently the consumer will only send a prefetch for a partition after all the records from the previous fetch have been consumed. This can lead to suboptimal pipelining when max.poll.records is set very low since the processing latency for a small set of records may be small compared to the latency of a fetch. An improvement suggested by [~junrao] is to send the fetch anyway even if we have unprocessed data buffered, but delay reading it from the socket until that data has been consumed. Potentially the consumer can delay reading _any_ pending fetch until it is ready to be returned to the user, which may help control memory better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)