[
https://issues.apache.org/jira/browse/HDFS-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092924#comment-18092924
]
ASF GitHub Bot commented on HDFS-17948:
---------------------------------------
qijiale76 opened a new pull request, #8579:
URL: https://github.com/apache/hadoop/pull/8579
### Description of PR
This PR improves how FedBalance builds the DistCp commands it submits.
Previously, FedBalance used a mostly fixed DistCp option set. This patch
adds operator-facing controls for several DistCp settings:
- Add `-skipAclPreserve` to disable ACL preservation.
- Add `hdfs.fedbalance.preserve.acl.enabled` to control ACL preservation
from configuration.
- Preserve ACLs only when both source and destination filesystems support
ACLs.
- Add `-preserveTimes` to preserve file timestamps.
- Add `-distcpStrategy` to pass a DistCp copy strategy.
- Add `-numListstatusThreads` to pass DistCp listStatus thread count.
- Register `-diffThreshold` in the shared FedBalance CLI options.
- Serialize the new options in `FedBalanceContext` so recovered jobs use the
same DistCp settings.
- Update FedBalance documentation and tests.
The default behavior remains conservative: timestamp preservation is
disabled unless requested, DistCp strategy and listStatus thread count use
DistCp defaults unless specified, and ACL preservation remains enabled only
when ACLs are supported on both sides.
### How was this patch tested?
Ran the related FedBalance unit tests.
This change has also been validated on a production cluster.
### AI Tooling
Contains content generated by Codex and Claude Code.
> Improve FedBalance DistCp option handling
> -----------------------------------------
>
> Key: HDFS-17948
> URL: https://issues.apache.org/jira/browse/HDFS-17948
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: QI, Jiale
> Priority: Minor
>
> FedBalance currently submits DistCp jobs with a fixed set of DistCp preserve
> options and does not expose several useful DistCp knobs through the
> FedBalance CLI.
> This makes FedBalance less flexible for operators. For example:
> - ACL preservation is always requested by the generated DistCp command, which
> can fail or be undesirable when ACLs are not enabled on one side.
> - File timestamp preservation cannot be requested explicitly.
> - DistCp copy strategy cannot be selected through FedBalance.
> - DistCp listStatus parallelism cannot be tuned through FedBalance.
> - `-diffThreshold` is documented but not registered in the shared CLI options.
> This issue proposes adding FedBalance options for these DistCp settings while
> keeping the existing default behavior as much as possible:
> - Add `-skipAclPreserve` to disable ACL preservation.
> - Preserve ACLs only when both source and destination filesystems support
> ACLs.
> - Add `hdfs.fedbalance.preserve.acl.enabled` as a configuration-level ACL
> preserve switch.
> - Add `-preserveTimes` to include file timestamp preservation.
> - Add `-distcpStrategy` to pass a DistCp copy strategy.
> - Add `-numListstatusThreads` to tune DistCp listStatus threads.
> - Register `-diffThreshold` in the shared CLI option set.
> - Persist the new options in `FedBalanceContext` so recovered jobs keep the
> same behavior.
> Defaults should remain conservative: ACL preservation stays enabled when
> supported by both sides, timestamp preservation remains disabled unless
> requested, and DistCp strategy/listStatus thread settings use DistCp defaults
> unless explicitly configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
