[
https://issues.apache.org/jira/browse/HUDI-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-5465:
--------------------------------------
Description:
lets say c50 is latest DC in MDT. c49 from DT comes through(multi-writer).
triggers compaction in MDT(since ignoring c49 there are no other pending
instants in DT). new base instant time is c50. and we add 49.deltacommit to
MDT. and during the process we crash.
rollback for 49 kicks in DT. When applying rollback of 49 to MDT, we detect 49
has already been compacted since last compacted time is 50 and {*}fail the
rollback when we try to apply to MDT{*}.
We need to fix this entire flow for rollbacks and compaction related
multi-writer scenarios.
Impact:
writes to MDT might fail at some point and users have to disable MDT and make
progress
was:
lets say c50 is latest DC in MDT. c49 from DT comes through(multi-writer).
triggers compaction in MDT(since ignoring c49 there are no other pending
instants in DT). new base instant time is c50. and we add 49.deltacommit to
MDT. and during the process we crash.
rollback for 49 kicks in DT. When applying rollback of 49 to MDT, we detect 49
has already been compacted since last compacted time is 50 and {*}fail the
rollback when we try to apply to MDT{*}.
We need to fix this entire flow for rollbacks and compaction related
multi-writer scenarios.
> Fix compaction and rollback handling in MDT for multi-writer scenarios in DT
> ----------------------------------------------------------------------------
>
> Key: HUDI-5465
> URL: https://issues.apache.org/jira/browse/HUDI-5465
> Project: Apache Hudi
> Issue Type: Bug
> Components: metadata
> Reporter: sivabalan narayanan
> Priority: Blocker
> Fix For: 0.13.0
>
>
> lets say c50 is latest DC in MDT. c49 from DT comes through(multi-writer).
> triggers compaction in MDT(since ignoring c49 there are no other pending
> instants in DT). new base instant time is c50. and we add 49.deltacommit to
> MDT. and during the process we crash.
> rollback for 49 kicks in DT. When applying rollback of 49 to MDT, we detect
> 49 has already been compacted since last compacted time is 50 and {*}fail the
> rollback when we try to apply to MDT{*}.
>
> We need to fix this entire flow for rollbacks and compaction related
> multi-writer scenarios.
>
> Impact:
> writes to MDT might fail at some point and users have to disable MDT and make
> progress
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
