Hi Maung, If your required.acks is 1 then the producer only ensures that one broker receives the data before it's sucessfully returned to the client.
Therefore if the broker crashes and lost all the data then you lose data, or similarly it can happen even before the data is fsynced. To ensure there are more copies of your data in case of failure scenarios you want to increase your required.acks to more than 1 to tolerate failuries. Also async producer doesn't wait until the data is sent before it returns, as it buffers and writes asynchronously. To ensure each write that has a succesful response is written you want to use the sync producer. Tim On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_t...@apple.com> wrote: > Hi, > > We are seeing less data on the brokers than we send form the producers: 84 > GB to 58 GB. > > What is the best way to ensure / detect if all data has been send properly to > the brokers from the producers. > > Is there any logs that we can check on the producers? > > Configuration is 5 Brokers, 2 producers, no replication factor, async and ask > is 1 and no compression. > > Thanks, > Maung
