yihua commented on code in PR #8782:
URL: https://github.com/apache/hudi/pull/8782#discussion_r1202556828
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java:
##########
@@ -199,7 +201,7 @@ protected boolean
isPendingCompactionScheduledForFileId(HoodieFileGroupId fgId)
protected void resetPendingCompactionOperations(Stream<Pair<String,
CompactionOperation>> operations) {
// Build fileId to Pending Compaction Instants
this.fgIdToPendingCompaction =
createFileIdToPendingCompactionMap(operations.map(entry ->
- Pair.of(entry.getValue().getFileGroupId(), Pair.of(entry.getKey(),
entry.getValue()))).collect(Collectors.toMap(Pair::getKey, Pair::getValue)));
+ Pair.of(entry.getValue().getFileGroupId(), Pair.of(entry.getKey(),
entry.getValue()))).collect(Collectors.toConcurrentMap(Pair::getKey,
Pair::getValue)));
Review Comment:
nit: in `HoodieTableFIleSystemView`, we have methods like
`createFileIdToBootstrapBaseFileMap`, `createFileIdToReplaceInstantMap`, etc.
Some existing methods including `createFileIdToReplaceInstantMap` have the
logic of instantiating concurrent hash map there. Should we just override the
implementation of `createFileIdToBootstrapBaseFileMap` instead of changing the
logic here?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java:
##########
@@ -199,7 +201,7 @@ protected boolean
isPendingCompactionScheduledForFileId(HoodieFileGroupId fgId)
protected void resetPendingCompactionOperations(Stream<Pair<String,
CompactionOperation>> operations) {
// Build fileId to Pending Compaction Instants
this.fgIdToPendingCompaction =
createFileIdToPendingCompactionMap(operations.map(entry ->
- Pair.of(entry.getValue().getFileGroupId(), Pair.of(entry.getKey(),
entry.getValue()))).collect(Collectors.toMap(Pair::getKey, Pair::getValue)));
+ Pair.of(entry.getValue().getFileGroupId(), Pair.of(entry.getKey(),
entry.getValue()))).collect(Collectors.toConcurrentMap(Pair::getKey,
Pair::getValue)));
}
Review Comment:
Yes, multiple partitions can concurrently update the file system view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
