Hi Damian, I was thinking of cases when there is bit-rot on the storage itself and we get a malformed record that cannot be de-serialized. There is an interesting intersection here with CRCs in both Kafka (already there, they throw on deserialization) and potentially local storage (we don't have CRCs here on the data files, though RocksDB has them on its write-ahead log records).
Basically in a nutshell, I'm saying that every deserialization exception should go through this new path. The user can decide to fail or continue. We could start with just poison pills from Kafka though and punt the storage one to later. Eno > On 26 May 2017, at 16:59, Damian Guy <damian....@gmail.com> wrote: > > Eno, > > Under what circumstances would you get a deserialization exception from the > state store? I can only think of the case where someone has provided a bad > deserializer to a method that creates a state store. In which case it would > be a user error and probably should just abort? > > Thanks, > Damian > > On Fri, 26 May 2017 at 16:32 Eno Thereska <eno.there...@gmail.com> wrote: > >> See latest reply to Jan's note. I think I unnecessarily broadened the >> scope of this KIP to the point where it sounded like it handles all sorts >> of exceptions. The scope should be strictly limited to "poison pill" >> records for now. Will update KIP, >> >> Thanks >> Eno >>> On 26 May 2017, at 16:16, Matthias J. Sax <matth...@confluent.io> wrote: >>> >>> "bad" for this case would mean, that we got an >>> `DeserializationException`. I am not sure if any other processing error >>> should be covered? >>> >>> @Eno: this raises one one question. Might it be better to allow for two >>> handlers instead of one? One for deserialization exception and one for >>> all other exceptions from user code? >>> >>> Just a thought. >>> >>> >>> -Matthias >>> >>> On 5/26/17 7:49 AM, Jim Jagielski wrote: >>>> >>>>> On May 26, 2017, at 5:13 AM, Eno Thereska <eno.there...@gmail.com> >> wrote: >>>>> >>>>> >>>>>> >>>>>> >>>>>> With regard to `DeserializationException`, do you thing it might make >>>>>> sense to have a "dead letter queue" as a feature to provide >> out-of-the-box? >>>>> >>>>> We could provide a special topic where bad messages go to, and then >> we'd have to add a config option for the user to provide a topic. Is that >> what you're thinking? >>>>> >>>> >>>> For various definitions of "bad"?? >>>> >>> >> >>
