Hi all, Just to update my previous comments:
After reading about Sarama and checking how Heka uses it, I would say the behaviour observed (message loss during failure) is by design. https://github.com/Shopify/sarama/issues/294#issuecomment-130693063 I have opened issue #1750 and I'm keen to try to fix it but wonder what would be the preferred approach: 1. Rework the current async kafka calls into synchronous Kafka calls 2. Create a separate output plugin using synchronous Sarama calls. Cheers On Fri, Sep 25, 2015 at 11:58 PM, Andre <[email protected]> wrote: > All, > > Just to clarify (as my previous message didn't mention this): > > During test 2 messages get lost in transit once the broker is shutdown > and despite the very low volume of messages 5 messages per second over > 25 seconds. > > Cheers > > On Fri, Sep 25, 2015 at 11:49 PM, Andre <[email protected]> wrote: >> Hi there, >> >> Is anyone able to confirm what is the expected behaviour of using the >> Kafka output in case of an unlikely multi-broker outage? >> >> I have been doing some tests against my pipeline (heka TcpInput -> >> KafkaOutput -> Broker) all running on the same machine and I noticed >> the following behavior: >> >> Test 1 - Broker is dead when heka starts: >> >> Error making runner for TestKafkaOutput: Initialization failed for >> 'TestKafkaOutput': kafka: client has run out of available brokers to >> talk to (Is your cluster reachable?) >> >> >> Test 2 - All brokers (1 in this case) go offline (healthy shutdown) >> during message streaming >> >> >> 2015/09/25 23:12:12 Plugin 'TestKafkaOutput' error: kafka error >> 'kafka: client has run out of available brokers to talk to (Is your >> cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost >> localprg[1234]: seq: 0000670524, runid: 1443184763, >> ' >> 2015/09/25 23:12:13 Plugin 'TestKafkaOutput' error: kafka error >> 'kafka: client has run out of available brokers to talk to (Is your >> cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost >> localprg[1234]: seq: 0000670525, runid: 1443184763, >> ' >> 2015/09/25 23:12:14 Plugin 'TestKafkaOutput' error: kafka error >> 'kafka: client has run out of available brokers to talk to (Is your >> cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost >> localprg[1234]: seq: 0000670526, runid: 1443184763 >> ' >> 2015/09/25 23:12:14 Plugin 'TestKafkaOutput' error: kafka error >> 'circuit breaker is open' for message '<38>2015-09-25T22:39:36 >> localhost localprg[1234]: seq: 0000670527, runid: 1443184763 >> >> >> I was surprised by this one. After reading the documentation I would >> assume that required_acks default WaitForLocal would prevent this from >> happening? >> >> Relevant TOML section is: >> >> [TestKafkaOutput] >> type = "KafkaOutput" >> message_matcher = "Type == 'TcpInput'" >> topic = "test_topic" >> addrs = ["localhost:9092"] >> encoder = "PayloadEncoder" >> >> >> >> >> I thank you in advance _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
