Kevin Tseng created FLINK-33479: ----------------------------------- Summary: Add Rerouting support to KafkaSink Key: FLINK-33479 URL: https://issues.apache.org/jira/browse/FLINK-33479 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Affects Versions: 1.18.0 Reporter: Kevin Tseng
KafkaSink currently operates in a fail / success model. If it fails it will throw FlinkRuntimeException, any record that may cause failure has to be filtered by the operators preceeding KafkaSink. If there's unexpected failure that was caught, this will create continuous failure until a hotfix has been provided and issue addressed. Relying on checkpoint to restore the process also create duplicated records produced within the interval, unless EXACTLY_ONCE is utilized. Since Flink is meant to be a real-time streaming process, any downtime can create detrimental effect to the downstream consumer of the data. I'm proposing to add an error handling injection/intercept to KafkaSink (user can define an error handler or route the error causing record to another downstream operator) to keep the process running. Only throwing RuntimeException when the interceptor was not provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)