[
https://issues.apache.org/jira/browse/HUDI-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706768#comment-17706768
]
Danny Chen commented on HUDI-2458:
----------------------------------
With the restriction aboove, here is the illustration:
*Proposal*
* relax MDT pending instants deduction to compact to ignore compaction and
clustering from DT;
* Ensure compaction of MDT ignores uncommitted data;
* this might work only for spurious deletes ignored flow.
*The new workflow*
1. t5: assumes there is requested compaction at t5 on DT timeline:
DT: t5.compaction.req
2. t10: the MDT compaction triggers (ignore the pending t5 on DT)
MDT: t10_002.commit
3. t20: DT triggers a ROLLBACK for t5
DT: t20.rb
MDT: t20.dc
We may not know if t5 was already committed to MDT or not.
And hence had to go for -F records. (only works for spurious deletes ignore
flow).
4. t30: t5 is retried and succeed in MDT but failed in DT (attempt1)
MDT: t5.dc with files f10, f11.
DT: crash
5. t31: MDT compaction triggered
MDT: t31_002.commit
The rollback payload for t5 in MDT decides that t5 is already synced. We do -F.
6. t40: t5 is retried and filed to commit to MDT (attempt2)
DT: t5.compaction.req._2nd with files f20, f21.
crash
The rollback payload for t5 in MDT decides that t5 is not synced. We do -F with
tombstone messages (only works for spurious deletes ignore flow))
> Relax compaction in metadata being fenced based on inflight requests in data
> table
> ----------------------------------------------------------------------------------
>
> Key: HUDI-2458
> URL: https://issues.apache.org/jira/browse/HUDI-2458
> Project: Apache Hudi
> Issue Type: Improvement
> Components: metadata
> Reporter: sivabalan narayanan
> Assignee: Ethan Guo
> Priority: Blocker
> Fix For: 0.13.1, 0.12.3
>
>
> Relax compaction in metadata being fenced based on inflight requests in data
> table.
> Compaction in metadata is triggered only if there are no inflight requests in
> data table. This might cause liveness problem since for very large
> deployments, we could either have compaction or clustering always in
> progress. So, we should try to see how we can relax this constraint.
>
> Proposal to remove this dependency:
> With recent addition of spurious deletes config, we can actually get away
> with this.
> As of now, we have 3 inter linked nuances.
> - Compaction in metadata may not kick in, if there are any inflight
> operations in data table.
> - Rollback when being applied to metadata table has a dependency on last
> compaction instant in metadata table. We might even throw exception if
> instant being rolledback is < latest metadata compaction instant time.
> - Archival in data table is fenced by latest compaction in metadata table.
>
> So, just incase data timeline has any dangling inflght operation (lets say
> someone tried clustering, and killed midway and did not ever attempt again),
> metadata compaction will never kick in at all for good. I need to check what
> does archival do for such inflight operations in data table though when it
> tries to archive near by commits.
>
> So, with spurious deletes support which we added recently, all these can be
> much simplified.
> Whenever we want to apply a rollback commit, we don't need to take different
> actions based on whether the commit being rolled back is already committed to
> metadata table or not. Just go ahead and apply the rollback. Merging of
> metadata payload records will take care of this. If the commit was already
> synced, final merged payload may not have spurious deletes. If the commit
> being rolledback was never committed to metadata, final merged payload may
> have some spurious deletes which we can ignore.
> With this, compaction in metadata does not need to have any dependency on
> inflight operations in data table.
> And we can loosen up the dependency of archival in data table on metadata
> table compaction as well.
> So, in summary, all the 3 dependencies quoted above will be moot if we go
> with this approach. Archival in data table does not have any dependency on
> metadata table compaction. Rollback when being applied to metadata table does
> not care about last metadata table compaction. Compaction in metadata table
> can proceed even if there are inflight operations in data table.
>
> Especially our logic to apply rollback metadata to metadata table will become
> a lot simpler and is easy to reason about.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
