lokeshj1703 opened a new pull request, #19243: URL: https://github.com/apache/hudi/pull/19243
### Describe During MDT (Metadata Table) initialization, zero-size data files can be included in the filenameToSizeMap in DirectoryInfo, which could result in initialization job failures. This change adds a new config hoodie.metadata.skip.zero.size.files.on.initialize (default false) that, when enabled, skips files with length == 0 while listing the data table during MDT initialization and logs a warning. A skipped_zero_size_files_on_initialize metric is emitted with the total count of skipped files across all partitions. Closes #18612 ### Impact Opt-in via config; default behavior is unchanged. When enabled, reinitialize jobs that previously failed due to zero-size data files being included in MDT will succeed. The skipped_zero_size_files_on_initialize metric surfaces the count of skipped files. ### Risk level (write none, low medium or high below) Low. The change only affects the MDT initialization file-listing path and is gated behind a config that defaults to off. ### Documentation Update None required beyond the new advanced config, which is self-documented. ### Contributor's checklist - [x] Read through contributor's guide (https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
