QI, Jiale created HDFS-17948:
--------------------------------
Summary: Improve FedBalance DistCp option handling
Key: HDFS-17948
URL: https://issues.apache.org/jira/browse/HDFS-17948
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: QI, Jiale
FedBalance currently submits DistCp jobs with a fixed set of DistCp preserve
options and does not expose several useful DistCp knobs through the FedBalance
CLI.
This makes FedBalance less flexible for operators. For example:
- ACL preservation is always requested by the generated DistCp command, which
can fail or be undesirable when ACLs are not enabled on one side.
- File timestamp preservation cannot be requested explicitly.
- DistCp copy strategy cannot be selected through FedBalance.
- DistCp listStatus parallelism cannot be tuned through FedBalance.
- `-diffThreshold` is documented but not registered in the shared CLI options.
This issue proposes adding FedBalance options for these DistCp settings while
keeping the existing default behavior as much as possible:
- Add `-skipAclPreserve` to disable ACL preservation.
- Preserve ACLs only when both source and destination filesystems support ACLs.
- Add `hdfs.fedbalance.preserve.acl.enabled` as a configuration-level ACL
preserve switch.
- Add `-preserveTimes` to include file timestamp preservation.
- Add `-distcpStrategy` to pass a DistCp copy strategy.
- Add `-numListstatusThreads` to tune DistCp listStatus threads.
- Register `-diffThreshold` in the shared CLI option set.
- Persist the new options in `FedBalanceContext` so recovered jobs keep the
same behavior.
Defaults should remain conservative: ACL preservation stays enabled when
supported by both sides, timestamp preservation remains disabled unless
requested, and DistCp strategy/listStatus thread settings use DistCp defaults
unless explicitly configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
