[
https://issues.apache.org/jira/browse/HUDI-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Sumit updated HUDI-2459:
------------------------------
Fix Version/s: 0.13.0
(was: 0.12.0)
> Support async compaction for metadata table
> -------------------------------------------
>
> Key: HUDI-2459
> URL: https://issues.apache.org/jira/browse/HUDI-2459
> Project: Apache Hudi
> Issue Type: Task
> Components: writer-core
> Reporter: sivabalan narayanan
> Assignee: Ethan Guo
> Priority: Blocker
> Fix For: 0.13.0
>
>
> For now, metadata table has inline compaction. But we need to come up with a
> strategy to support async compaction.
> Since MDT compaction is fenced based on inflight requests in datatable, if
> for some reason, compaction in data table kept on failing and never succeeds,
> we will never compact metadata table as well. This might turn out to be
> detrimental.
> So, we should come up with a strategy to support async compaction in metadata
> table.
>
> Some nuances:
> If there are delta commits corresponding to rollback, we should ensure the
> final state in base table reflect that and don't miss out on any details.
> F1 added with dc1. f1 removed by dc3 (rollback in datatable) and again f2
> (added by same commit instant) when retried in datatable (compaction for eg).
> So, final state should reflect just F2 being added and F1 in deleted state,
> irrespective of whether compaction is complete or not.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
