[
https://issues.apache.org/jira/browse/HUDI-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450766#comment-17450766
]
Prashant Wason commented on HUDI-2458:
--------------------------------------
Another way to solve this issue is to think in terms of instant completion
times. Currently, HUDI does not record time for completion of instants. But if
that was available then all of these conditions may be met.
Assume the following dataset timeline with long running clustering op (Metadata
table is not compacted due to the t2.replacecommit inflight):
t1.commit t2.replacecommit.inflight t3.commit
When t2 completes (say at time t4), it created t2.replacecommit. But what if
there was a way to track that t2 completed at t4? Then metadata table could be
tracking the completion times and compaction can proceed.
One way is to change the instant format to <start_time>-<end_time>.<action>
Another way is to check modifcation times of instant files (may not work across
all cloud stores).
> Relax compaction in metadata being fenced based on inflight requests in data
> table
> ----------------------------------------------------------------------------------
>
> Key: HUDI-2458
> URL: https://issues.apache.org/jira/browse/HUDI-2458
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
> Fix For: 0.11.0
>
>
> Relax compaction in metadata being fenced based on inflight requests in data
> table.
> Compaction is metadata is triggered only if there are no inflight requests in
> data table. This might cause liveness problem since for very large
> deployments, we could either have compaction or clustering always in
> progress. So, we should try to see how we can relax this constraint.
>
> Proposal to remove this dependency:
> With recent addition of spurious deletes config, we can actually get away
> with this.
> As of now, we have 3 inter linked nuances.
> - Compaction in metadata may not kick in, if there are any inflight
> operations in data table.
> - Rollback when being applied to metadata table has a dependency on last
> compaction instant in metadata table. We might even throw exception if
> instant being rolledback is < latest metadata compaction instant time.
> - Archival in data table is fenced by latest compaction in metadata table.
>
> So, just incase data timeline has any dangling inflght operation (lets say
> someone tried clustering, and killed midway and did not ever attempt again),
> metadata compaction will never kick in at all for good. I need to check what
> does archival do for such inflight operations in data table though when it
> tries to archive near by commits.
>
> So, with spurious deletes support which we added recently, all these can be
> much simplified.
> Whenever we want to apply a rollback commit, we don't need to take different
> actions based on whether the commit being rolled back is already committed to
> metadata table or not. Just go ahead and apply the rollback. Merging of
> metadata payload records will take care of this. If the commit was already
> synced, final merged payload may not have spurious deletes. If the commit
> being rolledback was never committed to metadata, final merged payload may
> have some spurious deletes which we can ignore.
> With this, compaction in metadata does not need to have any dependency on
> inflight operations in data table.
> And we can loosen up the dependency of archival in data table on metadata
> table compaction as well.
> So, in summary, all the 3 dependencies quoted above will be moot if we go
> with this approach. Archival in data table does not have any dependency on
> metadata table compaction. Rollback when being applied to metadata table does
> not care about last metadata table compaction. Compaction in metadata table
> can proceed even if there are inflight operations in data table.
>
> Especially our logic to apply rollback metadata to metadata table will become
> a lot simpler and is easy to reason about.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
