Yue Zhang created HUDI-2435:
-------------------------------
Summary: Tuning clustering job handle errors
Key: HUDI-2435
URL: https://issues.apache.org/jira/browse/HUDI-2435
Project: Apache Hudi
Issue Type: Task
Reporter: Yue Zhang
Before clustering job/ async clustering finished, hudi will perform errors
check using JavaRDD<WriteStatus> writeResponse
It is a collect spark action, when executor is crashed by accident and the
cache of JavaRDD<WriteStatus> is lost, so that this collect action will trigger
a complete compute and create unexpected marker files or data files.
We should use Option<HoodieCommitMetadata> commitMetadata to do handle errors
action instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
