sivabalan narayanan created HUDI-2476:
-----------------------------------------
Summary: Fix retried compaction commit in datatable fails when
applied to metadata w/ sync updates
Key: HUDI-2476
URL: https://issues.apache.org/jira/browse/HUDI-2476
Project: Apache Hudi
Issue Type: Improvement
Components: Writer Core
Reporter: sivabalan narayanan
Compaction and clustering has a static instant time. So, when retried it may
not have a new instant time, but the same.
So, lets walk through the scenario of what happens when compaction fails after
synced to metadata table.
c1, c2, cc3( compaction commit), c4.
c1, c2, c4 on completion is applied to metadata table.
cc3 also gets synced to metadata table, but before committing to data table, it
failed(process crashed). Its is a small window, but still a possibility.
So, from a timeline standpoint this is what looks like
data timeline:
c1 complete, c2 complete, cc3 inflight. c4 complete.
metadata timeline:
dc_c1, dc_c2, dc_cc3, dc_c4
Lets say there are few more commits went in.
data timeline:
c1 complete, c2 complete, cc3 inflight. c4 complete, c5 complete. c6 complete.
metadata timeline:
dc_c1, dc_c2, dc_cc3, dc_c4, dc_c5, dc_c6
Now, compaction in datatable is being re-attempted. So, first we rollback
pending compaction in data table. So, this will trigger an upsert to metadata
table. even thought this is a rollback, all updates to metadata table is an
upsert which would result in a delta table.
And then, the compaction will be retried in datatable. when this is nearing
completion, we try to upsert to metadata table. which will fail. because
already we have a completed dc_cc3 in metadata table.
Fix:
when a commit is being retried, we delete the completed instant and then
proceed with upsert. So, when log blocks/files are merged together, final state
will be intact and will ensure only those files added in 2nd attempt is
returned and those added during 1st attempt is not returned (since there will
be complimenting log block corresponding to a rollback).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
