Re: Message getting lost in Kafka + Spark Streaming

Thanks Sidney for your response, To check if all the messages are processed I used accumulator and also add a print statement for debuging. *val accum = ssc.sparkContext.accumulator(0, "Debug Accumulator")* *...* *...* *...* *val mappedDataStream = dataStream.map(_._2);* *

RE: Message getting lost in Kafka + Spark Streaming

Are you sure that every message gets processed? It could be that some messages failed passing the decoder. And during the processing, are you maybe putting the events into a map? That way, events with the same key could override each other and that way you'll have less final events.

Re: Message getting lost in Kafka + Spark Streaming

First thing I noticed, you should be using a singleton kafka producer, not recreating one every partition. It's threadsafe. On Tue, May 30, 2017 at 7:59 AM, Vikash Pareek wrote: > I am facing an issue related to spark streaming with kafka, my use case is as >