Thanks. I am using console-producer with following settings with lz4 broker compression codec 1. None producer compression codec 2. Snappy producer compression codec 3. lz4 producer compression codec
I send a 354 Byte message with each of the above settings. However, I do not see any kind of double compression happening, when the producer and broker compression codecs are different *Output 1:* offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 3693540371 payload: xxxx compressed 354Byte message using --deep-iteration offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 4190573446 payload: 354 byte message *Output 2:*offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 4075439033 payload: compressed 354B message using --deep-iteration offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 4061704088 payload: Same 354B message *Output 3:* offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 981370250 payload: compressed 354B message using --deep-iteration offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 468622988 payload: same 354B message Please note the compression codecs in the --deep-iteration case, Case 1 is OK, but in case 2 shouldn't it be SnappyCompression and 3 may be LZ4Compression Or is it visible when the message batch into large batches? Thanks Uddhav On Wed, Feb 14, 2018 at 6:05 PM, Manikumar <manikumar.re...@gmail.com> wrote: > If the broker "compression.type" is "producer", then the broker retains > the original compression codec set by the producer. > If the producer and broker codecs are different, then broker recompress > the data using broker "compression.type". > > On Wed, Feb 14, 2018 at 10:58 AM, Uddhav Arote <aroteudd...@gmail.com> > wrote: > > > Hi Kafka users, > > > > I am trying to understand the behavior of compression in Kafka. Consider > a > > scenario, where producer sets compression.codec "snappy" and broker's > > compression.code "lz4"? > > In this scenario, what is the behavior of the compression? > > > > As far as I have understood is the following, > > The messages compressed by the producer are wrapped in the wrapper > message > > and send to the broker. If the broker compression.codec is "producer", > the > > message is written as is to the log. > > In the code, > > > > https://github.com/apache/kafka/blob/962bc638f9c2ab249e5008a587ee78 > > e3ba35fcb9/core/src/main/scala/kafka/log/LogValidator.scala#L218 > > > > > > what I understand is that if the producer and broker codecs are not same, > > then the compression should happen again. > > > > But I am not sure about this. Can somebody tell me how this works? > > > > Thanks, > > Uddhav > > >
