[
https://issues.apache.org/jira/browse/HUDI-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2435:
--------------------------------------
Status: In Progress (was: Open)
> Tuning clustering job handle errors
> -----------------------------------
>
> Key: HUDI-2435
> URL: https://issues.apache.org/jira/browse/HUDI-2435
> Project: Apache Hudi
> Issue Type: Task
> Reporter: Yue Zhang
> Priority: Major
> Labels: pull-request-available
>
> Before clustering job/ async clustering finished, hudi will perform errors
> check using JavaRDD<WriteStatus> writeResponse
> It is a collect spark action, when executor is crashed by accident and the
> cache of JavaRDD<WriteStatus> is lost, so that this collect action will
> trigger a complete compute and create unexpected marker files or data files.
> We should use Option<HoodieCommitMetadata> commitMetadata to do handle errors
> action instead.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
