[
https://issues.apache.org/jira/browse/HUDI-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-3178:
--------------------------------------
Status: Resolved (was: Patch Available)
> Metadata table compaction can include invalid updates from failed actions on
> dataset
> ------------------------------------------------------------------------------------
>
> Key: HUDI-3178
> URL: https://issues.apache.org/jira/browse/HUDI-3178
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Prashant Wason
> Assignee: sivabalan narayanan
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.10.1
>
>
> Metadata Table v2 performs an inline compaction once a deltacommit has been
> written.
> Timeline:
> (on dataset) t1.commit.requested
> (on dataset) t1.commit.inflight
> ---- all parquet writes complete here, WriteStatus generated---
> (on metadata table) t1.deltacommit.requested
> (on metadata table) t1.deltacommit.inflight
> (on metadata table) t1.deltacommit
> ---- deltcommit completed ----
> (on metadata table) t1-001.compaction.requested
> (on metadata table) t1-001.compaction.inflight
> (on metadata table) t1-001.commit
> If the t1.commit fails on the dataset then metadata table has already
> included information from the t1.commit in its base files which will be
> returned to readers. The metadata table reader logic only checks for
> deltacommits against completed instants on the dataset timeline and assumes a
> base file is always SANE.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
