[
https://issues.apache.org/jira/browse/HADOOP-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HADOOP-16083:
--------------------------------
Attachment: HADOOP-16083.001.patch
Status: Patch Available (was: Open)
> DistCp shouldn't always overwrite the target file when checksums match
> ----------------------------------------------------------------------
>
> Key: HADOOP-16083
> URL: https://issues.apache.org/jira/browse/HADOOP-16083
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
> Affects Versions: 3.1.1, 3.2.0, 3.3.0
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HADOOP-16083.001.patch
>
>
> {code:java|title=CopyMapper#setup}
> ...
> try {
> overWrite = overWrite ||
> targetFS.getFileStatus(targetFinalPath).isFile();
> } catch (FileNotFoundException ignored) {
> }
> ...
> {code}
> The above code overrides config key "overWrite" to "true" when the target
> path is a file. Therefore, unnecessary transfer happens when the source and
> target file have the same checksums.
> My suggestion is: remove the code above. If the user insists to overwrite,
> just add -overwrite in the options:
> {code:bash|title=DistCp command with -overwrite option}
> hadoop distcp -overwrite hdfs://localhost:64464/source/5/6.txt
> hdfs://localhost:64464/target/5/6.txt
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
