wombatu-kun commented on code in PR #11065:
URL: https://github.com/apache/hudi/pull/11065#discussion_r1576188482
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/ManifestFileWriter.java:
##########
@@ -85,16 +85,22 @@ public synchronized void writeManifestFile(boolean
useAbsolutePath) {
public static Stream<String>
fetchLatestBaseFilesForAllPartitions(HoodieTableMetaClient metaClient,
boolean useFileListingFromMetadata, boolean useAbsolutePath) {
try {
- List<String> partitions = FSUtils.getAllPartitionPaths(new
HoodieLocalEngineContext(metaClient.getHadoopConf()),
- metaClient.getBasePath(), useFileListingFromMetadata);
- LOG.info("Retrieve all partitions: " + partitions.size());
-
Configuration hadoopConf = metaClient.getHadoopConf();
HoodieLocalEngineContext engContext = new
HoodieLocalEngineContext(hadoopConf);
HoodieMetadataFileSystemView fsView = new
HoodieMetadataFileSystemView(engContext, metaClient,
metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants(),
HoodieMetadataConfig.newBuilder().enable(useFileListingFromMetadata).build());
- return partitions.parallelStream().flatMap(partition ->
fsView.getLatestBaseFiles(partition).map(useAbsolutePath ?
HoodieBaseFile::getPath : HoodieBaseFile::getFileName));
+ Stream<HoodieBaseFile> allLatestBaseFiles;
+ if (useFileListingFromMetadata) {
+ LOG.info("Fetching all base files from MDT.");
+ allLatestBaseFiles = fsView.getLatestBaseFiles();
Review Comment:
Yes, you are right, `fsView.getLatestBaseFiles()` only returns already
loaded file-groups in the view. But I don't see any other approach to load all
latest files in one call to
`HoodieMetadataFileSystemView`/`HoodieTableMetadata`. It would be great if you
or @nsivabalan (as reporter of this task) give me some advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
