[
https://issues.apache.org/jira/browse/FLINK-33479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783869#comment-17783869
]
Bo Shen commented on FLINK-33479:
---------------------------------
I have encountered a similar situation in JdbcSink a while ago, the job could
fail and cannot recover when attempted to process a record that is too big to
write to a mysql table.
In my situation, I modified Jdbc connector to skip write errors and emits error
messages through logging system (plugged in a kafka appender, so the error
could be consumed by external systems).
I am fairly new to flink, and not sure if this approach violates any design
philosophy.
It would be nice if the commnuity have a proper way of managing error tolerance
in sinks from a more general perspective.
> Add Rerouting support to KafkaSink
> ----------------------------------
>
> Key: FLINK-33479
> URL: https://issues.apache.org/jira/browse/FLINK-33479
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: 1.18.0
> Reporter: Kevin Tseng
> Priority: Minor
>
> KafkaSink currently operates in a fail / success model. If it fails it will
> throw FlinkRuntimeException, any record that may cause failure has to be
> filtered by the operators preceeding KafkaSink. If there's unexpected failure
> that was caught, this will create continuous failure until a hotfix has been
> provided and issue addressed.
> Relying on checkpoint to restore the process also create duplicated records
> produced within the interval, unless EXACTLY_ONCE is utilized.
>
> Since Flink is meant to be a real-time streaming process, any downtime can
> create detrimental effect to the downstream consumer of the data.
> I'm proposing to add an error handling injection/intercept to KafkaSink (user
> can define an error handler or route the error causing record to another
> downstream operator) to keep the process running. Only throwing
> RuntimeException when the interceptor was not provided.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
