[
https://issues.apache.org/jira/browse/HDFS-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093115#comment-18093115
]
ASF GitHub Bot commented on HDFS-17949:
---------------------------------------
qijiale76 opened a new pull request, #8582:
URL: https://github.com/apache/hadoop/pull/8582
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
This PR adds an optional FedBalance verification phase after the final
DistCp succeeds.
The verification is disabled by default. When enabled, FedBalance compares
the source and destination ContentSummary directory count, file count, and
length after excluding snapshot content. A mismatch fails the job, while
transient ContentSummary read failures are retried by the procedure scheduler.
This provides a lightweight built-in verification mechanism without changing
the default FedBalance behavior.
### How was this patch tested?
Ran FedBalance unit tests locally and validated on a production cluster.
### AI Tooling
Contains content generated by Codex.
> Add optional FedBalance content summary verification
> ----------------------------------------------------
>
> Key: HDFS-17949
> URL: https://issues.apache.org/jira/browse/HDFS-17949
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: QI, Jiale
> Priority: Major
>
> FedBalance currently finishes after the final DistCp succeeds, but it does
> not provide an optional built-in verification step to compare the source and
> destination tree at a coarse content-summary level.
> This issue proposes adding an optional FedBalance verification phase after
> the final DistCp succeeds. The verification is disabled by default and can be
> enabled through CLI or configuration. When enabled, it compares the source
> and destination ContentSummary directory count, file count, and length after
> excluding snapshot content.
> This gives operators a lightweight built-in correctness check without
> changing the default FedBalance behavior.
> Proposed changes:
> - Add an optional VERIFY stage to DistCpProcedure.
> - Add a FedBalanceVerifier helper based on ContentSummary.
> - Add a CLI option and configuration key to enable verification.
> - Add tests for successful verification and mismatch detection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
