At LinkedIn, we only buffer messages in memory in the producer. We try as much as we can to make the Kafka cluster always available (with replication). The "on failure" feature you mentioned can be implemented by providing a callback. You can see the example in the javadoc.
*send <http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord, kafka.clients.producer.Callback)>*(ProducerRecord<http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html> record, Callback<http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html> callback) Thanks, Jun On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry <mistry.p.bhav...@gmail.com>wrote: > We are using Kafka for operation metrics and we do not want to loose any > data at all if there is issue with Network or all brokers needs to be > rebooted or operation reason down time while all producers are still > producing data on front end side. We use async to publish messages and we > are using Kafka version 0.8.0. > > > > Has any one implemented buffering on local disk (on producer side) and > transmit messages when network connection is restored? How do I get > handle to list of messages async thread could not transfer after x reties > ? I know new producer API has callback interface, but is per message not > per producer instance ? > > > > Is this final new Producer API ? > > http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html > > > is there a plan to add method like. onFailure(List<Messages> messages, > Exception exception ) ? > > > > > Basically, I have to address the QOS on producer side, and be able to > buffer on disk and retransmit all message to partitions that are reserved > for messages that happened in past. > > > > How does Linked-in handle QOS on producer side ? > > > Is there any plan to add this QOS feature on Producer Side with Strategies > to > store and retransmit the message ? If we do get the list of messages is > call back, will it be compressed data ? > > > > I would appreciate Kafka Developers and others feedback on how to implement > QOS. > > > Thanks, > > > Bhavesh >
