hudi-bot opened a new issue, #15950:
URL: https://github.com/apache/hudi/issues/15950
In metadata writer code, if async indexing is enabled, we assume every
partition needs to be built out async. but users can choose to pick 2
partitions for inline and 2 for async. As of now, we don't have that
flexibility.
Code snippet from HoodieBackedTableMetadataWriter
{code:java}
// if metadata table exists, then check if any of the enabled partition
types needs to be initialized
// NOTE: It needs to be guarded by async index config because if that is
enabled then initialization happens through the index scheduler.
if (!dataWriteConfig.isMetadataAsyncIndex()) {
Set<String> inflightAndCompletedPartitions =
getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
LOG.info("Async metadata indexing enabled and following partitions already
initialized: " + inflightAndCompletedPartitions);
List<MetadataPartitionType> partitionsToInit =
this.enabledPartitionTypes.stream()
.filter(p ->
!inflightAndCompletedPartitions.contains(p.getPartitionPath()) &&
!MetadataPartitionType.FILES.equals(p))
.collect(Collectors.toList());
// if there are no partitions to initialize or there is a pending
operation, then don't initialize in this round
if (partitionsToInit.isEmpty() || anyPendingDataInstant(dataMetaClient,
inflightInstantTimestamp)) {
return;
}
String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
initTableMetadata(); // re-init certain flags in BaseTableMetadata
initializeEnabledFileGroups(dataMetaClient, createInstantTime,
partitionsToInit);
initialCommit(createInstantTime, partitionsToInit);
updateInitializedPartitionsInTableConfig(partitionsToInit);
} {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6205
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
