Andrey Neporada created KAFKA-3979:
--------------------------------------
Summary: Optimize memory used by replication process by using
adaptive fetch message size
Key: KAFKA-3979
URL: https://issues.apache.org/jira/browse/KAFKA-3979
Project: Kafka
Issue Type: Improvement
Components: replication
Affects Versions: 0.10.0.0
Reporter: Andrey Neporada
Current replication process fetches messages in replica.fetch.max.bytes-sized
chunks.
Since replica.fetch.max.bytes should be bigger than max.message.bytes for
replication to work, one can face big memory consumption for replication
process, especially for installations with big number of partitions.
Proposed solution is to try to fetch messages in smaller chunks (say
replica.fetch.base.bytes).
If we encounter message bigger than current fetch chunk, we increase chunk
(f.e. twofold) and retry. After replicating this bigger message, we shrunk
fetch chunk size back until it reaches replica.fetch.base.bytes
replica.fetch.base.bytes should be chosen big enough not to affect throughput
and to be bigger than most of messages.
However, it can be much less than replica.fetch.max.bytes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
