Bartosz Mikulski created FLINK-33783:
----------------------------------------
Summary: Add options to ignore parsing error in Kafka SQL Connector
Key: FLINK-33783
URL: https://issues.apache.org/jira/browse/FLINK-33783
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka, Table SQL / API
Reporter: Bartosz Mikulski
h1. Current state
If an unparsable event enters a Flink Kafka Source in SQL then the whole
application restarts. For JSON format there is a property that allows to ignore
unparsable events. Other formats, like `confluent-avro` etc don't support that.
h1. Desired state
We would like to ignore the parsing exception in Kafka Source in SQL regardless
of the format used. Additionally, a new metric should be introduced that
returns a count of unparsable events seen so far.
In the future there should be a Dead Letter Queue handling in SQL Sources
similar to Kafka Streams:
[https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/].
For now, the universal ignore with metric would be enough.
h1. Implementation
We already have an implementation for this case in the Flink Kafka Connector
and we would like to open a pull request for it. However, we created the issue
first as per this recommendation
[https://flink.apache.org/how-to-contribute/contribute-code/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
