bvaradar commented on code in PR #11065:
URL: https://github.com/apache/hudi/pull/11065#discussion_r1575593126
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/ManifestFileWriter.java:
##########
@@ -85,16 +85,22 @@ public synchronized void writeManifestFile(boolean
useAbsolutePath) {
public static Stream<String>
fetchLatestBaseFilesForAllPartitions(HoodieTableMetaClient metaClient,
boolean useFileListingFromMetadata, boolean useAbsolutePath) {
try {
- List<String> partitions = FSUtils.getAllPartitionPaths(new
HoodieLocalEngineContext(metaClient.getHadoopConf()),
- metaClient.getBasePath(), useFileListingFromMetadata);
- LOG.info("Retrieve all partitions: " + partitions.size());
-
Configuration hadoopConf = metaClient.getHadoopConf();
HoodieLocalEngineContext engContext = new
HoodieLocalEngineContext(hadoopConf);
HoodieMetadataFileSystemView fsView = new
HoodieMetadataFileSystemView(engContext, metaClient,
metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants(),
HoodieMetadataConfig.newBuilder().enable(useFileListingFromMetadata).build());
- return partitions.parallelStream().flatMap(partition ->
fsView.getLatestBaseFiles(partition).map(useAbsolutePath ?
HoodieBaseFile::getPath : HoodieBaseFile::getFileName));
+ Stream<HoodieBaseFile> allLatestBaseFiles;
+ if (useFileListingFromMetadata) {
+ LOG.info("Fetching all base files from MDT.");
+ allLatestBaseFiles = fsView.getLatestBaseFiles();
Review Comment:
Looks like fsView.getLatestBaseFiles() only returns already loaded
file-groups in the view so some partitions may not be loaded at all. Can you
check if this is your intended behavior ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
