Inder, Compression in Kafka is implemented at message set level. This means that it compresses n messages and represents that as one message. So for a compression-enabled topic, in the broker partition, each message has several messages wrapped inside.
It is true that compression is most effective on very large block sizes, like a MB or so. But even with message sets of size 40-50K, we see a compression ratio of ~3x on our data. This is using GZIP, you may see different results with Snappy. Thanks, Neha On Wed, Dec 7, 2011 at 8:29 AM, Inder Pall <inder.p...@gmail.com> wrote: > Folks, > > may be this has been asked/discussed before and i missed it. How do we do > compression in kafka (at record level)? > So it would like each line/record in the broker partiton would be > compressed. But our individual records aren't big in size do we really see > the value of compression > > -- > Thanks, > - Inder > Tech Platforms @Inmobi > Linkedin - http://goo.gl/eR4Ub >
