We could not find producer msg rate from the matrices in the JConsole — give us some pointers.
Also confirming that the reduction in data is due to Avro encoding and we are calculating what we send to producer rather than the output of serializer encoder. Thanks, Maung On Jun 3, 2014, at 10:47 PM, Maung Than <maung_t...@apple.com> wrote: > Thanks, Jun. > > Will check and get back.. > > We are converting JSON to Avro and that conversion is done by the custom > serializer. > > Our volume calculation on the producer side is based on the AVRO generic > record that is passed to the producer send method— not of the encoded data > output from the serializer that is what actual got send the Broker I believe. > > That could be the gap and I am testing now without the customer serializer > and seeing the two volumes are very close. That could be it!! > > Thanks, > Maung > > On Jun 3, 2014, at 7:22 PM, Jun Rao <jun...@gmail.com> wrote: > >> We have a metric on msg rate in both the producer and the broker. Could you >> see if they match? >> >> Thanks, >> >> Jun >> >> >> On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_t...@apple.com> wrote: >> >>> Hi, >>> >>> We are seeing less data on the brokers than we send form the producers: >>> 84 GB to 58 GB. >>> >>> What is the best way to ensure / detect if all data has been send properly >>> to the brokers from the producers. >>> >>> Is there any logs that we can check on the producers? >>> >>> Configuration is 5 Brokers, 2 producers, no replication factor, async and >>> ask is 1 and no compression. >>> >>> Thanks, >>> Maung >>> >
