shangxinli commented on code in PR #20333:
URL: https://github.com/apache/hudi/pull/20333#discussion_r4284240192
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -505,7 +526,11 @@ public List<StoragePathInfo> getFileList(HoodieStorage
storage, StoragePath part
.map(e -> {
// NOTE: Since we know that the Metadata Table's Payload is simply a
file-name we're
// creating Hadoop's Path using more performant unsafe variant
- return new StoragePathInfo(new StoragePath(partitionPath,
e.getKey()), e.getValue().getSize(),
+ // A registered file lives outside the table, so it resolves against
the directory it was
+ // registered with rather than against this table's partition path.
+ String sourcePartitionPath = e.getValue().getSourcePartitionPath();
+ StoragePath parent = sourcePartitionPath == null ? partitionPath :
new StoragePath(sourcePartitionPath);
Review Comment:
Confirmed, and worse: a registered partition has no directory under the base
path at all, so the whole partition gets dropped, not just its files. Not
reachable until the writer lands — tracking it as a blocker there. @nsivabalan
where should it live?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/index/files/FilesIndexer.java:
##########
@@ -189,6 +192,9 @@ private static List<HoodieRecord>
convertMetadataToFilesPartitionRecords(HoodieC
// of the sizes as reported after every write, since
file-sizes are
// monotonically increasing (ie file-size never goes down,
unless deleted)
updatedFilesToSizesMapping.merge(fileName,
stat.getFileSizeInBytes(), Math::max);
+ if (stat.getSourcePartitionPath() != null) {
+ fileToSourcePartitionPath.put(fileName,
stat.getSourcePartitionPath());
Review Comment:
Added, though not the snippet — `put`'s return can't tell "first time" from
"seen with a null path", so it now compares seen names with `Objects.equals`.
The authoritative check goes in the registration planner.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2462,8 +2462,23 @@ public static Map<String, HoodieMetadataFileInfo>
combineFileSystemMetadata(Hood
}
return null;
}
+ // Where a file lives never changes while it exists under the
same name, so the two
+ // records can only ever carry the same source partition path,
and taking whichever
+ // one is set is commutative. Two different paths means the name
was reused for a
+ // file elsewhere, which ordering cannot resolve either: pick
the same one whichever
+ // way the records arrive, so readers merging in different
orders at least agree.
+ String oldSourcePartitionPath =
oldFileInfo.getSourcePartitionPath();
+ String newSourcePartitionPath =
newFileInfo.getSourcePartitionPath();
+ String sourcePartitionPath =
Review Comment:
Keeping it inline — it needs the file name for the warning and belongs
beside the size and is-deleted rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
