danny0405 commented on code in PR #18829:
URL: https://github.com/apache/hudi/pull/18829#discussion_r3426243253
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkInsertOverwriteCommitActionExecutor.java:
##########
@@ -101,6 +103,26 @@ protected List<String> getAllExistingFileIds(String
partitionPath) {
return
table.getSliceView().getLatestFileSlices(partitionPath).map(FileSlice::getFileId).distinct().collect(Collectors.toList());
}
+ @Override
+ protected Set<HoodieFileGroupId>
getFileGroupsBeingReplaced(HoodieData<HoodieRecord<T>> inputRecords) {
+ String staticOverwritePartition =
config.getStringOrDefault(HoodieInternalConfig.STATIC_OVERWRITE_PARTITION_PATHS);
+ List<String> partitionPaths;
+
+ if (StringUtils.nonEmpty(staticOverwritePartition)) {
+ // Static insert overwrite: use the configured partitions
+ partitionPaths = Arrays.asList(staticOverwritePartition.split(","));
+ } else {
+ // Dynamic insert overwrite: determine partitions from input records
+ partitionPaths =
inputRecords.map(HoodieRecord::getPartitionPath).distinct().collectAsList();
+ }
+
+ // Get all file groups in the partitions to be overwritten
+ return partitionPaths.stream()
+ .flatMap(partitionPath ->
table.getSliceView().getLatestFileSlices(partitionPath)
Review Comment:
can use the method `getAllExistingFileIds`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
