Exactly, that is the reason. To avoid that, in Spark 1.3 to-be-released, we have added a new Kafka API (called direct stream) which does not use Zookeeper at all to keep track of progress, and maintains offset within Spark Streaming. That can guarantee all records being received exactly-once. Its experimental for now, but we will make it stable. Please try it out.
TD On Mon, Feb 23, 2015 at 9:41 PM, V Dineshkumar <[email protected] > wrote: > Hi, > > My spark streaming application is pulling data from Kafka.To prevent data > loss I have implemented WAL and enable checkpointing.On killing my > application and restarting it I am able to prevent data loss now but > however I am getting duplicate messages. > > Is it because the application got killed before it was able checkpoint the > current processing state?? > If yes how to tackle the duplicate messages? > > Thanks, > Dinesh >
