[
https://issues.apache.org/jira/browse/HUDI-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2476:
--------------------------------------
Fix Version/s: 0.10.0
> Fix retried compaction commit in datatable fails when applied to metadata w/
> sync updates
> -----------------------------------------------------------------------------------------
>
> Key: HUDI-2476
> URL: https://issues.apache.org/jira/browse/HUDI-2476
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
> Fix For: 0.10.0
>
>
> Compaction and clustering has a static instant time. So, when retried it may
> not have a new instant time, but the same.
> So, lets walk through the scenario of what happens when compaction fails
> after synced to metadata table.
> c1, c2, cc3( compaction commit), c4.
> c1, c2, c4 on completion is applied to metadata table.
> cc3 also gets synced to metadata table, but before committing to data table,
> it failed(process crashed). Its is a small window, but still a possibility.
> So, from a timeline standpoint this is what looks like
> data timeline:
> c1 complete, c2 complete, cc3 inflight. c4 complete.
> metadata timeline:
> dc_c1, dc_c2, dc_cc3, dc_c4
> Lets say there are few more commits went in.
> data timeline:
> c1 complete, c2 complete, cc3 inflight. c4 complete, c5 complete. c6 complete.
> metadata timeline:
> dc_c1, dc_c2, dc_cc3, dc_c4, dc_c5, dc_c6
>
> Now, compaction in datatable is being re-attempted. So, first we rollback
> pending compaction in data table. So, this will trigger an upsert to metadata
> table. even thought this is a rollback, all updates to metadata table is an
> upsert which would result in a delta table.
> And then, the compaction will be retried in datatable. when this is nearing
> completion, we try to upsert to metadata table. which will fail. because
> already we have a completed dc_cc3 in metadata table.
>
> Fix:
> when a commit is being retried, we delete the completed instant and then
> proceed with upsert. So, when log blocks/files are merged together, final
> state will be intact and will ensure only those files added in 2nd attempt is
> returned and those added during 1st attempt is not returned (since there will
> be complimenting log block corresponding to a rollback).
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
