wombatu-kun commented on code in PR #11065:
URL: https://github.com/apache/hudi/pull/11065#discussion_r1577788385
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/ManifestFileWriter.java:
##########
@@ -85,16 +85,22 @@ public synchronized void writeManifestFile(boolean
useAbsolutePath) {
public static Stream<String>
fetchLatestBaseFilesForAllPartitions(HoodieTableMetaClient metaClient,
boolean useFileListingFromMetadata, boolean useAbsolutePath) {
try {
- List<String> partitions = FSUtils.getAllPartitionPaths(new
HoodieLocalEngineContext(metaClient.getHadoopConf()),
- metaClient.getBasePath(), useFileListingFromMetadata);
- LOG.info("Retrieve all partitions: " + partitions.size());
-
Configuration hadoopConf = metaClient.getHadoopConf();
HoodieLocalEngineContext engContext = new
HoodieLocalEngineContext(hadoopConf);
HoodieMetadataFileSystemView fsView = new
HoodieMetadataFileSystemView(engContext, metaClient,
metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants(),
HoodieMetadataConfig.newBuilder().enable(useFileListingFromMetadata).build());
- return partitions.parallelStream().flatMap(partition ->
fsView.getLatestBaseFiles(partition).map(useAbsolutePath ?
HoodieBaseFile::getPath : HoodieBaseFile::getFileName));
+ Stream<HoodieBaseFile> allLatestBaseFiles;
+ if (useFileListingFromMetadata) {
+ LOG.info("Fetching all base files from MDT.");
+ allLatestBaseFiles = fsView.getLatestBaseFiles();
Review Comment:
i've found `fsView.loadAllPartitions()` to load all partitions in one call,
and now all file-groups are loaded in the view before getting latest base files
by `fsView.getLatestBaseFiles()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
