[
https://issues.apache.org/jira/browse/FLINK-33783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794737#comment-17794737
]
Martijn Visser commented on FLINK-33783:
----------------------------------------
Thanks for opening this. I think this problem should preferably be solved
generically for all connectors and formats, not per format and connector. It
would be great if a FLIP can be opened for that
Sidenote: I would expect that you should not be able to end up in a situation
where you have unparsable events with confluent-avro. Are we talking about a
scenario where someone skipped the Schema Registry service and just wrote an
arbitrary byte stream (without magic byte) ?
> Add options to ignore parsing error in Kafka SQL Connector
> ----------------------------------------------------------
>
> Key: FLINK-33783
> URL: https://issues.apache.org/jira/browse/FLINK-33783
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Table SQL / API
> Reporter: Bartosz Mikulski
> Priority: Major
>
> h1. Current state
> If an unparsable event enters a Flink Kafka Source in SQL then the whole
> application restarts. For JSON format there is a property that allows to
> ignore unparsable events. Other formats, like `confluent-avro` etc don't
> support that.
> h1. Desired state
> We would like to ignore the parsing exception in Kafka Source in SQL
> regardless of the format used. Additionally, a new metric should be
> introduced that returns a count of unparsable events seen so far.
> In the future there should be a Dead Letter Queue handling in SQL Sources
> similar to Kafka Streams:
> [https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/].
> For now, the universal ignore with metric would be enough.
> h1. Implementation
> We already have an implementation for this case in the Flink Kafka Connector
> and we would like to open a pull request for it. However, we created the
> issue first as per this recommendation
> [https://flink.apache.org/how-to-contribute/contribute-code/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
