sivabalan narayanan created HUDI-6205:
-----------------------------------------
Summary: Users should be able to dictate which partition in MDT
needs to be built async and which one is inline
Key: HUDI-6205
URL: https://issues.apache.org/jira/browse/HUDI-6205
Project: Apache Hudi
Issue Type: Improvement
Components: metadata
Reporter: sivabalan narayanan
In metadata writer code, if async indexing is enabled, we assume every
partition needs to be built out async. but users can choose to pick 2
partitions for inline and 2 for async. As of now, we don't have that
flexibility.
Code snippet from HoodieBackedTableMetadataWriter
{code:java}
// if metadata table exists, then check if any of the enabled partition types
needs to be initialized
// NOTE: It needs to be guarded by async index config because if that is
enabled then initialization happens through the index scheduler.
if (!dataWriteConfig.isMetadataAsyncIndex()) {
Set<String> inflightAndCompletedPartitions =
getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
LOG.info("Async metadata indexing enabled and following partitions already
initialized: " + inflightAndCompletedPartitions);
List<MetadataPartitionType> partitionsToInit =
this.enabledPartitionTypes.stream()
.filter(p ->
!inflightAndCompletedPartitions.contains(p.getPartitionPath()) &&
!MetadataPartitionType.FILES.equals(p))
.collect(Collectors.toList());
// if there are no partitions to initialize or there is a pending operation,
then don't initialize in this round
if (partitionsToInit.isEmpty() || anyPendingDataInstant(dataMetaClient,
inflightInstantTimestamp)) {
return;
}
String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
initTableMetadata(); // re-init certain flags in BaseTableMetadata
initializeEnabledFileGroups(dataMetaClient, createInstantTime,
partitionsToInit);
initialCommit(createInstantTime, partitionsToInit);
updateInitializedPartitionsInTableConfig(partitionsToInit);
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
