Inder, Sorry for not being clear. Compression is done at the producer side only. The producer takes a batch of messages, compresses it into a message and sends it to the broker. Nothing changes on the broker side. It simply writes that compressed message to the partition.
The consumer reads a set of messages, detects the ones that are compressed messages, and transparently decompresses them before handing it over to the end user. Take a look at the CompressionUtils class and see how it is used. Thanks, Neha On Wed, Dec 7, 2011 at 9:41 AM, Inder Pall <inder.p...@gmail.com> wrote: > Neha, > > to ensure i get it right, how would this work...messages would be coming > from producer > and let's say there isn't any batching so they become available to the > consumer almost realtime. > Now does since kafka has to retain them till the retention period broker > partitions would be compressed. > > > > On Wed, Dec 7, 2011 at 10:03 PM, Neha Narkhede <neha.narkh...@gmail.com > >wrote: > > > Inder, > > > > Compression in Kafka is implemented at message set level. This means that > > it compresses n messages and represents that as one message. > > So for a compression-enabled topic, in the broker partition, each message > > has several messages wrapped inside. > > > > It is true that compression is most effective on very large block sizes, > > like a MB or so. But even with message sets of size 40-50K, we see a > > compression ratio of ~3x on our data. > > This is using GZIP, you may see different results with Snappy. > > > > Thanks, > > Neha > > > > > > On Wed, Dec 7, 2011 at 8:29 AM, Inder Pall <inder.p...@gmail.com> wrote: > > > > > Folks, > > > > > > may be this has been asked/discussed before and i missed it. How do we > do > > > compression in kafka (at record level)? > > > So it would like each line/record in the broker partiton would be > > > compressed. But our individual records aren't big in size do we really > > see > > > the value of compression > > > > > > -- > > > Thanks, > > > - Inder > > > Tech Platforms @Inmobi > > > Linkedin - http://goo.gl/eR4Ub > > > > > > > > > -- > Thanks, > - Inder > Tech Platforms @Inmobi > Linkedin - http://goo.gl/eR4Ub >
