Hi Carl, That sounds pretty frustrating; sorry about that.
I think I got a hint, but I'm not totally clear on the situation. It shouldn't be possible for data to get into the store if it can't be handled by the serde. There is a specific issue with global stores, but it doesn't sound like that's your situation. It sounds like what has happened is that the serde itself has "forgotten" how to handle data that it previously could handle, because someone deleted the schema from the registry. Realistically, this sounds more like a situation to prevent than one to handle. From an "ivory tower" perspective, it seems like you could: * establish a rule that once the schema has been used, it can only be evolved * or that you would establish a deprecation period in which both the new and old schema are present, so you can translate all the records * or if you really need to make a discontinuous change to the schema, it probably means that the store is now completely different in meaning, and you could just give it a different name and load it from scratch But if nothing like that fits the bill, then you could consider wrapping the avro serde in an error-handling serde that delegates to your avro serde, catches the deserialization exception, and instead returns a sentinel value that you can handle somehow? I hope this helps, Thanks, -John On Mon, Apr 20, 2020, at 21:59, Carl Graving wrote: > I have a state store being built from a stream with a custom set of value > serdes. In the stream processing I am able to handle errors appropriately > with the exception handler, but if I attempt to do a state store getAll and > use the iterator.hasNext() or next() or peek methods, any exceptions in the > deserialization are thrown. This is proving to be difficult to iterate over > the items in the state store and skip bad entries or remove them. I can see > different ways this could happen, such as avro schema removed from schema > registry (and cache) for an item in the state store. Am I missing an easy > way to deal with deserialization exceptions stemming from state store > iterator or gets? I will keep playing around, but it was hard to find a way > to reliably use the iterator (getAll) when hasNext, next, peekNext, etc all > throw exceptions. > > Thanks for any help or pointers on how to properly handle exceptions in > this case. > > Carl >
