All, I've put together the code to address this issue.
I basically followed the strategy used on TCP/HTTP/IRC output plugins together with Sarama's SyncProducer. https://github.com/trixpan/heka/commit/c5b455f80dd89216b3ff5d6da7880bb6522b6ee7 It works greatly and Heka is capable of surviving a full disconnect from the Kafka cluster without message losses but the impact to performance is huge. For systems with buffering that should matter much but those without would certainly feel the hit. I reckon the best approach would be leaving the current one untouched (for those who want performance), while introducing the code above as a SynchronousKafkaOutput module (for those who want durability). Keen to hear your opinions. _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
