[
https://issues.apache.org/jira/browse/FLINK-33783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798878#comment-17798878
]
Martijn Visser commented on FLINK-33783:
----------------------------------------
No worries [~bartoszdeepbi]
We shouldn't open up a PR yet, it's better to first have a discussion/FLIP on
the Dev mailing list. You can include your PoC for the solution in the proposal
of course, so that it would be rather easy to open up a PR at a later stage. If
you want, I can grant you FLIP permissions to create a page and then you can
open the discussion as well.
> Add options to ignore parsing error in Kafka SQL Connector
> ----------------------------------------------------------
>
> Key: FLINK-33783
> URL: https://issues.apache.org/jira/browse/FLINK-33783
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Table SQL / API
> Reporter: Bartosz Mikulski
> Priority: Major
>
> h1. Current state
> If an unparsable event enters a Flink Kafka Source in SQL then the whole
> application restarts. For JSON format there is a property that allows to
> ignore unparsable events. Other formats, like `confluent-avro` etc don't
> support that.
> h1. Desired state
> We would like to ignore the parsing exception in Kafka Source in SQL
> regardless of the format used. Additionally, a new metric should be
> introduced that returns a count of unparsable events seen so far.
> In the future there should be a Dead Letter Queue handling in SQL Sources
> similar to Kafka Streams:
> [https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/].
> For now, the universal ignore with metric would be enough.
> h1. Implementation
> We already have an implementation for this case in the Flink Kafka Connector
> and we would like to open a pull request for it. However, we created the
> issue first as per this recommendation
> [https://flink.apache.org/how-to-contribute/contribute-code/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
