1. Compression happens on the list of messages. 2. We compress a list of messages into a new message and the size of the latter has to be smaller than max.message.size and fetch.size in the consumer. 3. Yes, typically the bigger the list, the better the compression ratio. However, there is diminishing return once the list is big enough.
Thanks, Jun On Tue, Mar 12, 2013 at 10:10 AM, R S <mypostbo...@gmail.com> wrote: > I have question regarding GZIP compression. > > My env: I have enabled Compression.codec=1 essentially GZIP . Iam using > ProducerData api on the producer side, > We use following call in the code: > > producer.send(new ProducerData<Integer, String>( > topic, messageList)); > > messageList is list of messages , each message is a String. > No of messages in messageList is 2000 always. > Each message size is 1 KB . So total load in each call of producer.send is > (2000 * 1 KB) > > max.message.size is set to 3000 KB. > I use same value on the consumer side using fetch.size = 3000 KB . > > Question : > 1. Will compression compress each message in the list or , will it compress > whole of the list together as a single blob ? > 2. max.message.size .In kafka will max.message.size checks for each > message or does it checks with whole messageList size`? > 3. Having a bigger messagelist size will give me better compression?? > > -rahul >
