It is not double compression. When I say re-compression,  brokers
decompress the messages and compress again with
new codec.

On Wed, Feb 14, 2018 at 5:18 PM, Uddhav Arote <aroteudd...@gmail.com> wrote:

> Thanks.
>
> I am using console-producer with following settings with lz4 broker
> compression codec
> 1. None producer compression codec
> 2. Snappy producer compression codec
> 3. lz4 producer compression codec
>
> I send a 354 Byte message with each of the above settings. However, I do
> not see any kind of double compression happening, when the producer and
> broker compression codecs are different
>
> *Output 1:*
>  offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize:
> 61 magic: 1 compresscodec: LZ4CompressionCodec crc:  3693540371 payload:
> xxxx  compressed 354Byte message
>
>  using --deep-iteration
>  offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize:
> 354 magic: 1 compresscodec: NoCompressionCodec crc: 4190573446 payload: 354
> byte message
>
>
> *Output 2:*offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true
> payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 4075439033
> payload: compressed 354B message
>
> using --deep-iteration
> offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true
> payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 4061704088
> payload: Same 354B message
>
> *Output 3:*
> offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true
> payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 981370250
> payload: compressed 354B message
>
> using --deep-iteration
> offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true
> payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 468622988
> payload: same 354B message
>
> Please note the compression codecs in the --deep-iteration case,
>  Case 1 is OK, but in case 2 shouldn't it be SnappyCompression and 3 may be
> LZ4Compression
>
> Or is it visible when the message batch into large batches?
>
> Thanks
> Uddhav
>
> On Wed, Feb 14, 2018 at 6:05 PM, Manikumar <manikumar.re...@gmail.com>
> wrote:
>
> >   If the broker "compression.type" is "producer", then the broker retains
> > the original compression codec set by the producer.
> >   If the producer and broker codecs are different, then broker recompress
> > the data using broker "compression.type".
> >
> > On Wed, Feb 14, 2018 at 10:58 AM, Uddhav Arote <aroteudd...@gmail.com>
> > wrote:
> >
> > > Hi Kafka users,
> > >
> > > I am trying to understand the behavior of compression in Kafka.
> Consider
> > a
> > > scenario, where producer sets compression.codec "snappy" and broker's
> > > compression.code  "lz4"?
> > > In this scenario, what is the behavior of the compression?
> > >
> > > As far as I have understood is the following,
> > > The messages compressed by the producer are wrapped in the wrapper
> > message
> > > and send to the broker. If the broker compression.codec is "producer",
> > the
> > > message is written as is to the log.
> > > In the code,
> > >
> > > https://github.com/apache/kafka/blob/962bc638f9c2ab249e5008a587ee78
> > > e3ba35fcb9/core/src/main/scala/kafka/log/LogValidator.scala#L218
> > >
> > >
> > > what I understand is that if the producer and broker codecs are not
> same,
> > > then the compression should happen again.
> > >
> > > But I am not sure about this. Can somebody tell me how this works?
> > >
> > > Thanks,
> > > Uddhav
> > >
> >
>

Reply via email to