[
https://issues.apache.org/jira/browse/HDFS-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
QI, Jiale updated HDFS-17948:
-----------------------------
Priority: Major (was: Minor)
> Improve FedBalance DistCp option handling
> -----------------------------------------
>
> Key: HDFS-17948
> URL: https://issues.apache.org/jira/browse/HDFS-17948
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: QI, Jiale
> Priority: Major
> Labels: pull-request-available
>
> FedBalance currently submits DistCp jobs with a fixed set of DistCp preserve
> options and does not expose several useful DistCp knobs through the
> FedBalance CLI.
> This makes FedBalance less flexible for operators. For example:
> - ACL preservation is always requested by the generated DistCp command, which
> can fail or be undesirable when ACLs are not enabled on one side.
> - File timestamp preservation cannot be requested explicitly.
> - DistCp copy strategy cannot be selected through FedBalance.
> - DistCp listStatus parallelism cannot be tuned through FedBalance.
> - `-diffThreshold` is documented but not registered in the shared CLI options.
> This issue proposes adding FedBalance options for these DistCp settings while
> keeping the existing default behavior as much as possible:
> - Add `-skipAclPreserve` to disable ACL preservation.
> - Preserve ACLs only when both source and destination filesystems support
> ACLs.
> - Add `hdfs.fedbalance.preserve.acl.enabled` as a configuration-level ACL
> preserve switch.
> - Add `-preserveTimes` to include file timestamp preservation.
> - Add `-distcpStrategy` to pass a DistCp copy strategy.
> - Add `-numListstatusThreads` to tune DistCp listStatus threads.
> - Register `-diffThreshold` in the shared CLI option set.
> - Persist the new options in `FedBalanceContext` so recovered jobs keep the
> same behavior.
> Defaults should remain conservative: ACL preservation stays enabled when
> supported by both sides, timestamp preservation remains disabled unless
> requested, and DistCp strategy/listStatus thread settings use DistCp defaults
> unless explicitly configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
