Compression in Kafka

2018-02-13 Thread Uddhav Arote
Hi Kafka users, I am trying to understand the behavior of compression in Kafka. Consider a scenario, where producer sets compression.codec "snappy" and broker's compression.code "lz4"? In this scenario, what is the behavior of the compression? As far as I have understood is the following, The me

Re: Compression in Kafka

2018-02-14 Thread Uddhav Arote
ype". > > On Wed, Feb 14, 2018 at 10:58 AM, Uddhav Arote > wrote: > > > Hi Kafka users, > > > > I am trying to understand the behavior of compression in Kafka. Consider > a > > scenario, where producer sets compression.codec "snappy" and bro

Re: Compression in Kafka

2018-02-14 Thread Uddhav Arote
> decompress the messages and compress again with > new codec. > > On Wed, Feb 14, 2018 at 5:18 PM, Uddhav Arote > wrote: > > > Thanks. > > > > I am using console-producer with following settings with lz4 broker > > compression codec > > 1. None producer

Keeping track of ingest time of messages in pipeline (cluster 1-> mm -> cluster 2 -> ..)

2018-04-22 Thread Uddhav Arote
Hi, The V1 message format is 1. v1 (supported since 0.10.0) 2. Message => Crc MagicByte Attributes Key Value 3. Crc => int32 4. MagicByte => int8 5. Attributes => int8 6. Timestamp => int64 7. Key => bytes 8. Value => bytes Would it be a good suggestion to h

Re: Keeping track of ingest time of messages in pipeline (cluster 1-> mm -> cluster 2 -> ..)

2018-04-23 Thread Uddhav Arote
Any thoughts here? On 2018/04/23 05:47:24, Uddhav Arote wrote: > Hi,> > > The V1 message format is> > > >1. v1 (supported since 0.10.0)> >2. Message => Crc MagicByte Attributes Key Value> >3. Crc => int32> >4. MagicByte

Metric for follower's replication latency

2018-05-23 Thread Uddhav Arote
Hello all, I have a set of brokers and I want to know 'how much time do my brokers spend in replication of a message?'. I have two questions: 1. I did not find any available metric for replication latency (avg, max). 2. Why is there no metric for understanding broker's replication behavior? Uddha