Hi, My spark streaming application is pulling data from Kafka.To prevent data loss I have implemented WAL and enable checkpointing.On killing my application and restarting it I am able to prevent data loss now but however I am getting duplicate messages.
Is it because the application got killed before it was able checkpoint the current processing state?? If yes how to tackle the duplicate messages? Thanks, Dinesh
