Hi all, I was recently discussing deserialization errors and how they can be handled in Kafka consumers.
I believe the current code still throws an exception if deserialization fails which stops consumption unless you seek past that record. This creates an issue for a JDBC sink connector that uses Avro data for example, as this is effectively dead in the water when hitting a "poison pill" record. I've dug around for a little while and found the following issues on this: https://issues.apache.org/jira/browse/KAFKA-4740 https://issues.apache.org/jira/browse/KAFKA-5858 https://issues.apache.org/jira/browse/KAFKA-5211 mentions that this behavior was actually changed for a little while at some time in the past, but that change was snuck in without a KIP. There are also a few tickets specifically around Connect, but those refer more to the wording of the error message than the actual functionality. -- Is there a "gospel" way of handling this that people have found works for them? Or if not, would we want to extend the java consumer to allow configuring behavior for this case? At the very least it would be quite easy to add an option to simply ignore deserialization errors, thinking a bit further, we might also allow specifying a pluggable error handler that could be used to send malformed records back to a dead letter queue or something similar. Best regards, Sönke