Agreed! Once and only once is very difficult if not impossible. 2PC makes the window smaller still (but does not 100% eliminate it) at a significant performance cost.
On Fri, Aug 24, 2012 at 9:53 AM, Jun Rao <jun...@gmail.com> wrote: > Taylor, > > Yes, with ack, if you get an exception, it means that the messages may or > may not be successfully received by the broker. Producer can choose to > resend (which could introduce dups) or ignore (which could introduce loss). > Guaranteeing exactly once delivery is a bit hard. > > Thanks, > > Jun > > On Fri, Aug 24, 2012 at 9:05 AM, Taylor Gautier <tgaut...@tagged.com> > wrote: > > > Re-sending could lead to duplicated messages however - consider the case > > that the broker has committed the message and sent the ack but the ack is > > either sitting in the send buffer of the broker or recv buffer of the > > producer. > > > > Just saying that you will trade off a relatively small amount of loss > for a > > probably smaller amount of duplication using only a simple ack. > > > > > > On Fri, Aug 24, 2012 at 6:49 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > Xiaoyu, > > > > > > In 0.7, we have this problem that the producer doesn't receive any ack. > > So, > > > syncProducer.send is considered successful as soon as the messages are > in > > > the socket buffer. If the broker goes down before the socket buffer is > > > flushed, those supposedly successful messages are lost. What's worse is > > > that the producer doesn't know this since it doesn't wait for a > response. > > > Such lost messages should be small. However, I am not sure how to > > > reduce/prevent it. This issue will be addressed in 0.8, in which the > > > producer will receive an ack. If a broker goes down in the middle of a > > > send, the producer will get an exception and can resend. > > > > > > Thanks, > > > > > > Jun > > > > > > On Thu, Aug 23, 2012 at 5:19 PM, xiaoyu wang <xiaoyu.w...@gmail.com> > > > wrote: > > > > > > > Hello, > > > > > > > > We are using sync produce to push messages to kafka brokers. It will > > stop > > > > once it receives an IOException: connection reset by peer. It seems > > when > > > a > > > > broker goes down, we lose some messages. I have reduced > > > > "log.flush.interval" to 1, still see > 200 message loss. > > > > > > > > I also reduced the batch.size on producer side to 10, but the message > > > loss > > > > is about the same. > > > > > > > > So, what's the best way to minimize message loss on broker down? > > > > > > > > Thanks, > > > > > > > > -Xiaoyu > > > > > > > > > >
