[
https://issues.apache.org/jira/browse/HUDI-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8808:
------------------------------
Remaining Estimate: 2h
Original Estimate: 2h
> Fix concurrent execution of appending rollback blocks in the same file group
> ----------------------------------------------------------------------------
>
> Key: HUDI-8808
> URL: https://issues.apache.org/jira/browse/HUDI-8808
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: Y Ethan Guo
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.0.1
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> When there are multiple log files generated in the same file group in a
> inflight deltacommit, rollback of such a deltacommit can fail, because (1)
> the rollback plan contains multiple rollback requests targeting the log files
> to roll back in the same file group (2) concurrent execution of these
> rollback requests in Spark executors cause creation of new rollback log files
> in parallel, which involves determining the new log version concurrently,
> leading to the same new log version to be used in multiple executors, causing
> marker creation to fail (i.e., multiple executors try to create the marker on
> the same file name, and subsequent marker creation requests fail due to the
> fact that marker already exists).
> Note that this is for Hudi 0.x format only (backwards compatible writer).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
