[
https://issues.apache.org/jira/browse/HUDI-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470197#comment-17470197
]
Manoj Govindassamy commented on HUDI-3178:
------------------------------------------
Here are the proposals to address this problem:
# Move the Metadata table inline compaction just before the inflight commit -
This way the compaction will not include the current inflight commit and will
not be affected by the end result of this commit in the data table
# Move the Metadata table inline compaction and other table services after the
data table commit completion, but within the current/sam transaction - This way
metadata table table operations are guaranteed to see only the completed
commits for the latest one.
> Metadata table compaction can include invalid updates from failed actions on
> dataset
> ------------------------------------------------------------------------------------
>
> Key: HUDI-3178
> URL: https://issues.apache.org/jira/browse/HUDI-3178
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Prashant Wason
> Priority: Blocker
> Fix For: 0.10.1
>
>
> Metadata Table v2 performs an inline compaction once a deltacommit has been
> written.
> Timeline:
> (on dataset) t1.commit.requested
> (on dataset) t1.commit.inflight
> ---- all parquet writes complete here, WriteStatus generated---
> (on metadata table) t1.deltacommit.requested
> (on metadata table) t1.deltacommit.inflight
> (on metadata table) t1.deltacommit
> ---- deltcommit completed ----
> (on metadata table) t1-001.compaction.requested
> (on metadata table) t1-001.compaction.inflight
> (on metadata table) t1-001.commit
> If the t1.commit fails on the dataset then metadata table has already
> included information from the t1.commit in its base files which will be
> returned to readers. The metadata table reader logic only checks for
> deltacommits against completed instants on the dataset timeline and assumes a
> base file is always SANE.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
