Hi Boris, In Kafka 0.7, the producer does not get any ACK from the server, so the protocol is fire and forget. What could happen is if a broker is shutting down, the messages might still live in the producer's socket buffer and the server can shutdown before getting a chance to read the producer's data from the socket buffer. These messages will be lost silently.
Questions for you - 1. Were any of the brokers being shutdown/bounced during this time ? 2. If not, did you take a look at producer side trace logging that proves that the producer actually dispatched that data to the server ? Thanks Neha On Thu, Nov 29, 2012 at 2:22 AM, boris kogan <[email protected]> wrote: > Hi, All > > We encounter an issue with Kafka of messages loss: > > We have Kafka 0.7.0 installed in cluster mode on three nodes(on partition > per node configured). > > We are using kafka java API to send bulk of messages(1000 messages in each) > in sync mode. > > After the last bulk is sent we close the producer. > > We found that sometimes (~one in thousand) bulk do not arrive to kafka > without any error/exception/warning in logs. > > Did anyone encountered such issue or have an idea what is wrong? > > > > Thanks, > > Boris > > -- > This message may contain confidential and/or privileged information. > If you are not the addressee or authorized to receive this on behalf of the > addressee you must not use, copy, disclose or take action based on this > message or any information herein. > If you have received this message in error, please advise the sender > immediately by reply email and delete this message. Thank you.
