danny0405 commented on PR #20486: URL: https://github.com/apache/hudi/pull/20486#issuecomment-6675591392
@shangxinli Could you review the validation semantics change in this PR, particularly allowing inconclusive comparisons to proceed under `FAIL` with zero tolerance? The PR changes validation from **comparing the computed offset sum with records written** to **first determining whether that sum is exact or only a lower bound**. | Scenario | Current master | After this PR | | --- | --- | --- | | All partitions comparable | Require count equality within tolerance | Unchanged | | Added, missing, or reset partitions; written count above the known difference | Treat as a mismatch; apply tolerance and failure policy | Allow the commit; warn that validation is inconclusive | | Incomplete comparison; written count equals the known difference | Report validation passed (or skip as empty when both counts are zero) | Allow the commit; report inconclusive | | Incomplete comparison; written count below the known difference | Apply tolerance and failure policy | Same decision, described as a lower-bound deficit | | Checkpoints contain no partition offsets, e.g. Flink cluster-only metadata | Treat the expected count as zero | Treat the comparison as inconclusive; skip equality validation | | First commit, missing or invalid checkpoints | Skip validation | Unchanged | For example, with previous checkpoint `topic,0:100` and current checkpoint `topic,0:110,1:5`, the known difference is 10, but partition 1's starting offset is unknown. With `FAIL` and zero tolerance: - **15 records written:** previously rejected; now allowed with an inconclusive warning. - **10 records written:** previously reported as validated; now allowed but inconclusive. - **3 records written:** still rejected because the count is below the known lower bound. The key semantic change is that **an allowed commit does not imply a verified count match for incomplete comparisons**. The PR preserves detection of deficits below the known lower bound, but cannot establish full consistency or reject excess counts when some partition contributions are unknown. This extends the existing skip-on-unavailable-checkpoint behavior to incomplete partition comparisons and relaxes rejection for some commits. It would be helpful to confirm that this is the intended `FAIL` policy contract and whether the configuration documentation's “exact match required” wording should be clarified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
