We are using Kafka for operation metrics and we do not want to loose any data at all if there is issue with Network or all brokers needs to be rebooted or operation reason down time while all producers are still producing data on front end side. We use async to publish messages and we are using Kafka version 0.8.0.
Has any one implemented buffering on local disk (on producer side) and transmit messages when network connection is restored? How do I get handle to list of messages async thread could not transfer after x reties ? I know new producer API has callback interface, but is per message not per producer instance ? Is this final new Producer API ? http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html is there a plan to add method like. onFailure(List<Messages> messages, Exception exception ) ? Basically, I have to address the QOS on producer side, and be able to buffer on disk and retransmit all message to partitions that are reserved for messages that happened in past. How does Linked-in handle QOS on producer side ? Is there any plan to add this QOS feature on Producer Side with Strategies to store and retransmit the message ? If we do get the list of messages is call back, will it be compressed data ? I would appreciate Kafka Developers and others feedback on how to implement QOS. Thanks, Bhavesh
