Hi Deepak, It sounds like you're saying that the exception handler is correctly indicating that Streams should "Continue", and that if you stop the app after handling an exceptional record but before the next commit, Streams re-processes the record?
If that's what you're seeing, then it's how the system is designed to operate. We don't commit after every record because the overhead would be enormous. If you can't tolerate seeing duplicates in your error file, then it sounds like the simplest thing for you would be to maintain an index of the records you have already saved off so that you can gracefully handle re-processing. E.g., you might have a separate file per topic-partition that you update after appending to your error log to indicate the highest offset you've handled. Then, you can read it from the exception handler to see if the record you're handling is already logged. Just an idea. I hope this helps, -John On Tue, 2020-09-01 at 16:36 +0530, Deepak Raghav wrote: > Hi Team > > Just a reminder. > Can you please help me with this? > > Regards and Thanks > Deepak Raghav > > > > On Tue, Sep 1, 2020 at 1:44 PM Deepak Raghav <deepakragha...@gmail.com> > wrote: > > > Hi Team > > > > I have created a CustomExceptionHandler class by > > implementing DeserializationExceptionHandler interface to handle the > > exception during deserialization time. > > > > But the problem with this approach is that if there is some exception > > raised for some record and after that stream is stopped and > > restarted again, it reads those bad messages again. > > > > I am storing those bad messages in some file in the filesystem and with > > this approach, duplicate messages are getting appended in the file when the > > stream is started since those bad message's offset are not getting > > increased. > > > > Please let me know if I missed anything. > > > > Regards and Thanks > > Deepak Raghav > > > >
