nsivabalan commented on code in PR #18611:
URL: https://github.com/apache/hudi/pull/18611#discussion_r3559738940
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1076,7 +1076,9 @@ private List<DirectoryInfo>
listAllPartitionsFromFilesystem(String initializatio
final int fileListingParallelism =
metadataWriteConfig.getFileListingParallelism();
StorageConfiguration<?> storageConf = dataMetaClient.getStorageConf();
final String dirFilterRegex =
dataWriteConfig.getMetadataConfig().getDirectoryFilterRegex();
+ final boolean skipZeroSizeFiles =
dataWriteConfig.getMetadataConfig().shouldSkipZeroSizeFilesOnInitialize();
Review Comment:
Agreeing with the bot on this one — `listAllPartitionsFromFilesystem` is
also invoked from `update(HoodieRestoreMetadata, ...)` at line 1744, and
reading `skipZeroSizeFiles` from config here means the skip (and the
`skipped_zero_size_files_on_initialize` metric) fires during restore too. On
the restore path, a zero-size file that's already tracked in MDT but gets
filtered out of the FS listing will land in `filesDeleted` via
`fetchOutofSyncFilesRecordsFromMetadataTable` (around line 2321) — producing a
spurious MDT delete while the file is still on disk. That's a real correctness
issue, not just a naming/scope mismatch.
Please scope this to the initialize path only. Simplest fix is to pass
`skipZeroSizeFiles` as an explicit parameter to
`listAllPartitionsFromFilesystem` — pass `true` (gated by config) from the
initialize caller at line 431, and `false` unconditionally from the restore
caller at line 1744. That keeps the behavior consistent with what the PR/config
name promises (`on.initialize`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
