sivabalan narayanan created HUDI-7478:
-----------------------------------------
Summary: Fix max delta commits guard check w/ MDT
Key: HUDI-7478
URL: https://issues.apache.org/jira/browse/HUDI-7478
Project: Apache Hudi
Issue Type: Bug
Components: metadata
Reporter: sivabalan narayanan
protected static void checkNumDeltaCommits(HoodieTableMetaClient metaClient,
int maxNumDeltaCommitsWhenPending) \{
final HoodieActiveTimeline activeTimeline =
metaClient.reloadActiveTimeline();
Option<HoodieInstant> lastCompaction =
activeTimeline.filterCompletedInstants()
.filter(s -> s.getAction().equals(COMPACTION_ACTION)).lastInstant();
int numDeltaCommits = lastCompaction.isPresent()
?
activeTimeline.getDeltaCommitTimeline().findInstantsAfter(lastCompaction.get().getTimestamp()).countInstants()
: activeTimeline.getDeltaCommitTimeline().countInstants();
if (numDeltaCommits > maxNumDeltaCommitsWhenPending) {
throw new HoodieMetadataException(String.format("Metadata table's
deltacommits exceeded %d: "
+ "this is likely caused by a pending instant in the data table.
Resolve the pending instant "
+ "or adjust `%s`, then restart the pipeline.",
maxNumDeltaCommitsWhenPending,
HoodieMetadataConfig.METADATA_MAX_NUM_DELTACOMMITS_WHEN_PENDING.key()));
}
}
Here we account for action type "compaction. But compaction completed instant
will have "commit" as action. So, we need to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
