[
https://issues.apache.org/jira/browse/HUDI-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhenqiu Huang updated HUDI-8881:
--------------------------------
Summary: Potential dataloss in Flink hudi sink (was: Potentail dataloss in
Flink hudi sink)
> Potential dataloss in Flink hudi sink
> -------------------------------------
>
> Key: HUDI-8881
> URL: https://issues.apache.org/jira/browse/HUDI-8881
> Project: Apache Hudi
> Issue Type: Improvement
> Components: flink
> Reporter: Zhenqiu Huang
> Assignee: Zhenqiu Huang
> Priority: Critical
>
> Let's say checkpoint A is completed, then coodinator start to hudi commit.
> If there is a write error, then the table will be rolledback and flink job
> will fail after the exception is thrown. When the job restart from failure,
> what will happen? From my understanding, the time between kafka offset in
> checkpoint A -1 and offset in checkpoint A will lose.
> we should check whether WriteMetadataEvent contains write failure once the
> event is received, if any rollback immediately and throw exception to prevent
> checkpoint to complete when there is any write failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
