[
https://issues.apache.org/jira/browse/HUDI-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675724#comment-17675724
]
sivabalan narayanan commented on HUDI-5465:
-------------------------------------------
if a compaction instant time is c50 and later we have a DC w/ c49, if this DC
crashes, rollback of DC49 might fail.
code block in ListingBasedRollbackStrategy
{code:java}
// For sanity, log-file base-instant time can never be less than base-commit on
which we are rolling back
checkArgument(
HoodieTimeline.compareTimestamps(latestFileSlice.getBaseInstantTime(),
HoodieTimeline.LESSER_THAN_OR_EQUALS, rollbackInstant.getTimestamp()),
"Log-file base-instant could not be less than the instant being rolled
back"); {code}
With this context, can we revisit the workflow and ensure there are no gaps.
> Fix compaction and rollback handling in MDT for multi-writer scenarios in DT
> ----------------------------------------------------------------------------
>
> Key: HUDI-5465
> URL: https://issues.apache.org/jira/browse/HUDI-5465
> Project: Apache Hudi
> Issue Type: Bug
> Components: metadata
> Reporter: sivabalan narayanan
> Assignee: Ethan Guo
> Priority: Blocker
> Fix For: 0.13.0
>
>
> lets say c50 is latest DC in MDT. c49 from DT comes through(multi-writer).
> triggers compaction in MDT(since ignoring c49 there are no other pending
> instants in DT). new base instant time is c50. and we add 49.deltacommit to
> MDT. and during the process we crash.
> rollback for 49 kicks in DT. When applying rollback of 49 to MDT, we detect
> 49 has already been compacted since last compacted time is 50 and {*}fail the
> rollback when we try to apply to MDT{*}.
>
> We need to fix this entire flow for rollbacks and compaction related
> multi-writer scenarios.
>
> Impact:
> writes to MDT might fail at some point and users have to disable MDT and make
> progress
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
