deepakpanda93 opened a new pull request, #19497: URL: https://github.com/apache/hudi/pull/19497
### Describe the issue this Pull Request addresses Closes #15126 / [HUDI-3880](https://issues.apache.org/jira/browse/HUDI-3880). `HoodieMetadataTableValidator` throws as soon as the partition list from the file system and the partition list from the metadata table disagree: ```java if (misMatch.get()) { String message = "Compare Partitions Failed! ..."; log.error(message); throw new HoodieValidationException(message); } ``` This happens in `validatePartitions(...)`, before the caller gets a chance to run `validateFilesInPartition(...)` for any partition. So when the lists differ by a single partition, the run reports only that the lists differ and says nothing about whether the file listings for the other partitions are consistent. An operator triaging a validation failure has to fix the partition-level difference and re-run before learning whether anything else is wrong. Most of the original ticket has already landed — the validator now forgives extra file-system partitions that are empty, and metadata-table partitions belonging to a commit newer than the one being validated. The remaining gap is this one: no file listings are compared for the partitions the two sides *do* agree on. ### Summary and Changelog Users get a more complete picture from a single failing validation run: when the partition lists disagree, the report now also says whether the file listings for the partitions both sides agree on are consistent. - `validatePartitions(...)` now takes `baseFilesForCleaning` so it can run the same per-partition file validation the caller does. - On a genuine mismatch, the intersection of the two partition lists is computed and `validateFilesInPartition(...)` is run for each partition in it. The result is appended to the exception message. - The partition mismatch remains the reported cause and the validator still fails. The added comparison is supplementary, so if it cannot run — for example the metadata validation context fails to open — that is logged and noted in the message rather than allowed to mask the original mismatch. - When the lists are disjoint the message says so explicitly, so the output never implies something was checked when it was not. - Tests added to `TestHoodieMetadataTableValidator`: an end-to-end case on a real table (one partition removed from the file system, the other two compared), a disjoint-lists case, a case where the supplementary comparison itself fails, and a case where an extra empty file-system partition is not a mismatch at all. Example of the new output, from the end-to-end test: ``` Compare Partitions Failed! Additional 0 partitions from FS, but missing from MDT : "" and additional 1 partitions from MDT, but missing from FS listing : "2015/03/16". All 2 partitions from FS listing 2016/03/15,2015/03/17 File listings match for all 2 partitions common to FS listing and MDT. ``` ### Impact Validator-only. No change to the write or read path, and no change to whether a validation run passes or fails — only to how much a failing run tells you. The extra work is a per-partition file comparison that runs solely on the failure path, over the intersection of the two partition lists. ### Risk Level low Validator-only change confined to the failure path. The supplementary comparison is wrapped so that a failure in it cannot mask the partition mismatch that triggered it. ### Documentation Update None. No new config, public API, or user-facing behaviour change. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [x] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
