[
https://issues.apache.org/jira/browse/HUDI-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8808:
------------------------------
Description:
When there are multiple log files generated in the same file group in a
inflight deltacommit, rollback of such a deltacommit can fail, because (1) the
rollback plan contains multiple rollback requests targeting the log files to
roll back in the same file group (2) concurrent execution of these rollback
requests in Spark executors cause creation of new rollback log files in
parallel, which involves determining the new log version concurrently, leading
to the same new log version to be used in multiple executors, causing marker
creation to fail (i.e., multiple executors try to create the marker on the same
file name, and subsequent marker creation requests fail due to the fact that
marker already exists).
Note that this is for Hudi 0.x format only (backwards compatible writer).
> Fix concurrent execution of appending rollback blocks in the same file group
> ----------------------------------------------------------------------------
>
> Key: HUDI-8808
> URL: https://issues.apache.org/jira/browse/HUDI-8808
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.0.1
>
>
> When there are multiple log files generated in the same file group in a
> inflight deltacommit, rollback of such a deltacommit can fail, because (1)
> the rollback plan contains multiple rollback requests targeting the log files
> to roll back in the same file group (2) concurrent execution of these
> rollback requests in Spark executors cause creation of new rollback log files
> in parallel, which involves determining the new log version concurrently,
> leading to the same new log version to be used in multiple executors, causing
> marker creation to fail (i.e., multiple executors try to create the marker on
> the same file name, and subsequent marker creation requests fail due to the
> fact that marker already exists).
> Note that this is for Hudi 0.x format only (backwards compatible writer).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
