It should be sth like clientId-MessagesPerSec.
Thanks,
Jun
On Wed, Jun 4, 2014 at 9:35 AM, Maung Than wrote:
>
> We could not find producer msg rate from the matrices in the JConsole —
> give us some pointers.
>
> Also confirming that the reduction in data is due to Avro encoding and we
> are
We could not find producer msg rate from the matrices in the JConsole — give us
some pointers.
Also confirming that the reduction in data is due to Avro encoding and we are
calculating what we send to producer rather than the output of serializer
encoder.
Thanks,
Maung
On Jun 3, 2014, at 1
Thanks, Jun.
Will check and get back..
We are converting JSON to Avro and that conversion is done by the custom
serializer.
Our volume calculation on the producer side is based on the AVRO generic record
that is passed to the producer send method— not of the encoded data output from
the ser
Yes. We did..some output of it..
2014-06-03 21:46:09 INFO Producer:68 - Shutting down producer
2014-06-03 21:46:09 INFO ProducerSendThread:68 - Begin shutting down
ProducerSendThread
2014-06-03 21:46:09 INFO ProducerSendThread:68 - Shutdown ProducerSendThread
complete
2014-06-03 21:46:09 INFO
By the way if you're using async producer how do you verify that you
sent all the data from the producer?
Do you shutdown the producer before you check?
Tim
On Tue, Jun 3, 2014 at 3:27 PM, Maung Than wrote:
> Thanks, Tim.
>
> We are just trying to benchmark the kafka producers and there is no i
We have a metric on msg rate in both the producer and the broker. Could you
see if they match?
Thanks,
Jun
On Tue, Jun 3, 2014 at 2:13 PM, Maung Than wrote:
> Hi,
>
> We are seeing less data on the brokers than we send form the producers:
> 84 GB to 58 GB.
>
> What is the best way to ensure
Thanks, Tim.
We are just trying to benchmark the kafka producers and there is no issue with
cluster or brokers being down in this case.
We are seeing way less data on the borers after calculating the sizes of the
logs on the brokers) and there is no compression.
We send 84 GB, but total log
Hi Maung,
If your required.acks is 1 then the producer only ensures that one
broker receives the data before it's sucessfully returned to the
client.
Therefore if the broker crashes and lost all the data then you lose
data, or similarly it can happen even before the data is fsynced.
To ensure th
