[ https://issues.apache.org/jira/browse/HADOOP-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755424#comment-16755424 ]
Siyao Meng edited comment on HADOOP-16083 at 1/29/19 10:15 PM: --------------------------------------------------------------- [~ste...@apache.org] Thanks for the comment. The case involved here is *single file* copy with CRC check enabled. If the target path is a directory, *for HDFS* it will skip files that has the same checksum. For example, the command below will always copy and overwrite the file at the target. {code:bash|title=Target path is a file} $ hadoop distcp -update hdfs:///src/2.txt hdfs:///dst/2.txt ... Bytes Copied=6 Bytes Expected=6 Files Copied=1 {code} But this command wouldn't skip the copy if the target file exists and is the same, note the target path is a directory now instead of a file. {code:bash|title=Target path is a directory} $ hadoop distcp -update hdfs:///src/2.txt hdfs:///dst/ ... Bytes Skipped=6 Files Skipped=1 {code} I'm aware that the change might lead to subtle failures of other applications. For HDFS, so far I figured the only thing that changed with the patch would be the modification time of the file on target FS. This is tested in TestCopyMapper#testSingleFileCopy() and TestCopyMapperCompositeCrc#testSingleFileCopy(). Both test case failures would be addressed in rev 002. I agree that if the target FS 1. doesn't support checksum; or 2. disabled checksum; or 3. doesn't support timestamp modification (touch), the file should always be copied. What new tests should be added in AbstractContractDistCpTest? was (Author: smeng): [~ste...@apache.org] Thanks for the comment. The case involved here is *single file* copy with CRC check enabled. If the target path is a directory, *for HDFS* it will skip files that has the same checksum. For example, the command below will always copy and overwrite the file at the target. {code:bash|title=Target path is a file} $ hadoop distcp -update hdfs:///src/2.txt hdfs:///dst/2.txt ... Bytes Copied=6 Bytes Expected=6 Files Copied=1 {code} But this command wouldn't skip the copy is the target file exists and is the same, note the target path is a directory now instead of a file. {code:bash|title=Target path is a directory} $ hadoop distcp -update hdfs:///src/2.txt hdfs:///dst/ ... Bytes Skipped=6 Files Skipped=1 {code} I'm aware that the change might lead to subtle failures of other applications. For HDFS, so far I figured the only thing that changed with the patch would be the modification time of the file on target FS. This is tested in TestCopyMapper#testSingleFileCopy() and TestCopyMapperCompositeCrc#testSingleFileCopy(). Both test case failures would be addressed in rev 002. I agree that if the target FS 1. doesn't support checksum; or 2. disabled checksum; or 3. doesn't support timestamp modification (touch), the file should always be copied. What new tests should be added in AbstractContractDistCpTest? > DistCp shouldn't always overwrite the target file when checksums match > ---------------------------------------------------------------------- > > Key: HADOOP-16083 > URL: https://issues.apache.org/jira/browse/HADOOP-16083 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp > Affects Versions: 3.2.0, 3.1.1, 3.3.0 > Reporter: Siyao Meng > Assignee: Siyao Meng > Priority: Major > Attachments: HADOOP-16083.001.patch > > > {code:java|title=CopyMapper#setup} > ... > try { > overWrite = overWrite || > targetFS.getFileStatus(targetFinalPath).isFile(); > } catch (FileNotFoundException ignored) { > } > ... > {code} > The above code overrides config key "overWrite" to "true" when the target > path is a file. Therefore, unnecessary transfer happens when the source and > target file have the same checksums. > My suggestion is: remove the code above. If the user insists to overwrite, > just add -overwrite in the options: > {code:bash|title=DistCp command with -overwrite option} > hadoop distcp -overwrite hdfs://localhost:64464/source/5/6.txt > hdfs://localhost:64464/target/5/6.txt > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org