[
https://issues.apache.org/jira/browse/HUDI-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-5437:
--------------------------------------
Description:
In multi-writer scenario, we can have issues w/ the way we apply rollbacks.
c50 is latest DC in MDT. c49 from DT comes through. triggers compaction w/ 50
as base commit time. and while adding 49.deltacommit to MDT, we crash. rollback
for 49 kicks in DT. When we try to apply this rollback to DMT, we deduce as
though 49 is already compacted since latest base file instant is 50 and we
might fail the rollback when we try to apply to MDT. please revisit this logic
to verify the claim and fix it.
we could simplify how to apply rollback metadata if spurious deletes are
allowed. lets revisit this and see if it can be simplified for common user.
was:
In multi-writer scenario, we can have issues w/ the way we apply rollbacks.
c50 is latest DC in MDT. c49 from DT comes through. triggers compaction w/ 50
as base commit time. and while adding 49.deltacommit to MDT, we crash. rollback
for 49 kicks in DT. When we try to apply this rollback to DMT, we deduce as
though 49 is already compacted since latest base file instant is 50 and we
might fail the rollback when we try to apply to MDT. please revisit this logic
to verify the claim and fix it.
> Fix how we apply rollbacks to MDT
> ---------------------------------
>
> Key: HUDI-5437
> URL: https://issues.apache.org/jira/browse/HUDI-5437
> Project: Apache Hudi
> Issue Type: Bug
> Components: metadata
> Reporter: sivabalan narayanan
> Priority: Critical
> Fix For: 0.13.0
>
>
> In multi-writer scenario, we can have issues w/ the way we apply rollbacks.
>
> c50 is latest DC in MDT. c49 from DT comes through. triggers compaction w/ 50
> as base commit time. and while adding 49.deltacommit to MDT, we crash.
> rollback for 49 kicks in DT. When we try to apply this rollback to DMT, we
> deduce as though 49 is already compacted since latest base file instant is 50
> and we might fail the rollback when we try to apply to MDT. please revisit
> this logic to verify the claim and fix it.
>
> we could simplify how to apply rollback metadata if spurious deletes are
> allowed. lets revisit this and see if it can be simplified for common user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
