Yes. We did..some output of it.. 2014-06-03 21:46:09 INFO Producer:68 - Shutting down producer 2014-06-03 21:46:09 INFO ProducerSendThread:68 - Begin shutting down ProducerSendThread 2014-06-03 21:46:09 INFO ProducerSendThread:68 - Shutdown ProducerSendThread complete 2014-06-03 21:46:09 INFO ProducerPool:68 - Closing all sync producers
On Jun 3, 2014, at 9:58 PM, Timothy Chen <tnac...@gmail.com> wrote: > By the way if you're using async producer how do you verify that you > sent all the data from the producer? > > Do you shutdown the producer before you check? > > Tim > > On Tue, Jun 3, 2014 at 3:27 PM, Maung Than <maung_t...@apple.com> wrote: >> Thanks, Tim. >> >> We are just trying to benchmark the kafka producers and there is no issue >> with cluster or brokers being down in this case. >> >> We are seeing way less data on the borers after calculating the sizes of the >> logs on the brokers) and there is no compression. >> >> We send 84 GB, but total logs sizes are only 58 GB on the brokers. >> >> Since replication factor is zero, can we use ack other than 1? >> >> Maung >> >> On Jun 3, 2014, at 3:00 PM, Timothy Chen <tnac...@gmail.com> wrote: >> >>> Hi Maung, >>> >>> If your required.acks is 1 then the producer only ensures that one >>> broker receives the data before it's sucessfully returned to the >>> client. >>> >>> Therefore if the broker crashes and lost all the data then you lose >>> data, or similarly it can happen even before the data is fsynced. >>> >>> To ensure there are more copies of your data in case of failure >>> scenarios you want to increase your required.acks to more than 1 to >>> tolerate failuries. >>> >>> Also async producer doesn't wait until the data is sent before it >>> returns, as it buffers and writes asynchronously. To ensure each write >>> that has a succesful response is written you want to use the sync >>> producer. >>> >>> Tim >>> >>> On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_t...@apple.com> wrote: >>>> Hi, >>>> >>>> We are seeing less data on the brokers than we send form the producers: >>>> 84 GB to 58 GB. >>>> >>>> What is the best way to ensure / detect if all data has been send properly >>>> to the brokers from the producers. >>>> >>>> Is there any logs that we can check on the producers? >>>> >>>> Configuration is 5 Brokers, 2 producers, no replication factor, async and >>>> ask is 1 and no compression. >>>> >>>> Thanks, >>>> Maung >>
