danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234510320
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1689,40 +1804,75 @@ protected Pair<HoodieData<HoodieRecord>,
List<HoodieFileGroupId>> tagRecordsWith
try (HoodieTableFileSystemView fsView =
HoodieTableMetadataUtil.getFileSystemViewForMetadataTable(metadataMetaClient)) {
List<HoodieFileGroupId> hoodieFileGroupIdList = new ArrayList<>();
for (Map.Entry<String, HoodieData<HoodieRecord>> entry :
partitionRecordsMap.entrySet()) {
- final String partitionName = entry.getKey();
+ final String partitionPath = entry.getKey();
HoodieData<HoodieRecord> records = entry.getValue();
+ boolean isPartitionedRLI = Objects.equals(partitionPath,
RECORD_INDEX.getPartitionPath()) &&
dataWriteConfig.isPartitionedRecordIndexEnabled();
List<FileSlice> fileSlices =
-
HoodieTableMetadataUtil.getPartitionLatestFileSlices(metadataMetaClient,
Option.ofNullable(fsView), partitionName);
- if (fileSlices.isEmpty()) {
- // scheduling or initialising of INDEX only initializes the file
group and not add commit
- // so if there are no committed file slices, look for inflight slices
- ValidationUtils.checkState(isInitializing ||
dataMetaClient.getTableConfig().getMetadataPartitionsInflight().contains(partitionName),
- String.format("Partition %s should be part of inflight metadata
partitions here %s", partitionName,
dataMetaClient.getTableConfig().getMetadataPartitionsInflight()));
- fileSlices =
getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient,
Option.ofNullable(fsView), partitionName);
+
HoodieTableMetadataUtil.getPartitionLatestFileSlices(metadataMetaClient,
Option.ofNullable(fsView), partitionPath);
+ // scheduling of INDEX only initializes the file group and not add
commit
+ // so if there are no committed file slices, look for inflight slices
+ if (isPartitionedRLI || fileSlices.isEmpty()) {
+ // For isPartitionedRLI, new partitions added to the data table will
cause new filegroups that are not yet commited
+ // therefore, we always need to look for inflight filegroups
+ ValidationUtils.checkState(isInitializing ||
dataMetaClient.getTableConfig().getMetadataPartitionsInflight().contains(partitionPath)
|| isPartitionedRLI,
Review Comment:
can we simplify the logic to be more clear:
```java
if (fileSlices.isEmpty()) {
ValidationUtils.checkState(isInitializing ||
dataMetaClient.getTableConfig() ...;
fileSlices =
getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient,
Option.ofNullable(fsView), partitionName);
} else if (isPartitionedRLI) {
fileSlices =
getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient,
Option.ofNullable(fsView), partitionPath);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
