We are fronting all our Kafka requests with a simple web service (we do some additional massaging and writing to other stores as well). The new KafkaProducer in 0.8.2 seems very geared towards producer batching. Most of our payload are single messages.
Producer batching basically sets us up for lost messages if our web service goes down with unflushed messaged in the producer. Another issue is when we have a batch of records. It looks like I have to call producer.send for each record and deal with individual futures returned. Are there any patterns for primarily single message requests, without losing data? I understand the throughput will be low. Thanks! -Neelesh
