[
https://issues.apache.org/jira/browse/HUDI-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470247#comment-17470247
]
Prashant Wason commented on HUDI-3178:
--------------------------------------
The only difference between the two approaches seems to be whether the current
operation will be part of the compaction or not, right? So there is no huge
difference per-se and whichever is simpler to implement (wrt locking etc) is
preferable.
Lets also think about how async compaction will be achieved (in future) with
those two options. I already see the need for async compaction in my
record-level-index testing where one of my tables has a 280GB+ total
record_level_index partition and compacting it takes 1+hr.
In case of async compaction, the compactIfNecessary() code can schedule the
compaction but not perform it. The actual compaction can be performed in a
separate pipeline. This can be supported by both proposals.
So at the moment I cannot decide which is better proposal.
> Metadata table compaction can include invalid updates from failed actions on
> dataset
> ------------------------------------------------------------------------------------
>
> Key: HUDI-3178
> URL: https://issues.apache.org/jira/browse/HUDI-3178
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Prashant Wason
> Priority: Blocker
> Fix For: 0.10.1
>
>
> Metadata Table v2 performs an inline compaction once a deltacommit has been
> written.
> Timeline:
> (on dataset) t1.commit.requested
> (on dataset) t1.commit.inflight
> ---- all parquet writes complete here, WriteStatus generated---
> (on metadata table) t1.deltacommit.requested
> (on metadata table) t1.deltacommit.inflight
> (on metadata table) t1.deltacommit
> ---- deltcommit completed ----
> (on metadata table) t1-001.compaction.requested
> (on metadata table) t1-001.compaction.inflight
> (on metadata table) t1-001.commit
> If the t1.commit fails on the dataset then metadata table has already
> included information from the t1.commit in its base files which will be
> returned to readers. The metadata table reader logic only checks for
> deltacommits against completed instants on the dataset timeline and assumes a
> base file is always SANE.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
