Re: Kafka Streaming message loss

2016-11-21 Thread Michael Noll
Also: Since your testing is purely local, feel free to share the code you have been using so that we can try to reproduce what you're observing. -Michael On Mon, Nov 21, 2016 at 4:04 PM, Michael Noll wrote: > Please don't take this comment the wrong way, but have you

Re: Kafka Streaming message loss

2016-11-21 Thread Michael Noll
Please don't take this comment the wrong way, but have you double-checked whether your counting code is working correctly? (I'm not implying this could be the only reason for what you're observing.) -Michael On Fri, Nov 18, 2016 at 4:52 PM, Eno Thereska wrote: > Hi

Re: Kafka Streaming message loss

2016-11-18 Thread Eno Thereska
Hi Ryan, Perhaps you could share some of your code so we can have a look? One thing I'd check is if you are using compacted Kafka topics. If so, and if you have non-unique keys, compaction happens automatically and you might only see the latest value for a key. Thanks Eno > On 18 Nov 2016, at

Kafka Streaming message loss

2016-11-18 Thread Ryan Slade
Hi I'm trialling Kafka Streaming for a large stream processing job, however I'm seeing message loss even in the simplest scenarios. I've tried to boil it down to the simplest scenario where I see loss which is the following: 1. Ingest messages from an input stream (String, String) 2. Decode