[ https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682933#comment-17682933 ]
ASF GitHub Bot commented on HADOOP-18596: ----------------------------------------- mehakmeet commented on code in PR #5308: URL: https://github.com/apache/hadoop/pull/5308#discussion_r1092974460 ########## hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java: ########## @@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws Exception { verifyFileContents(localFS, dest, block); } + @Test + public void testDistCpUpdateCheckFileSkip() throws Exception { Review Comment: There is `TestHDFSContractDistCp.java` that does implement this suite. I have run it before and it works successfully. Let me know if we still need to add another one specifically for hdfs. > Distcp -update between different cloud stores to use modification time while > checking for file skip. > ---------------------------------------------------------------------------------------------------- > > Key: HADOOP-18596 > URL: https://issues.apache.org/jira/browse/HADOOP-18596 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp > Reporter: Mehakmeet Singh > Assignee: Mehakmeet Singh > Priority: Major > Labels: pull-request-available > > Distcp -update currently relies on File size, block size, and Checksum > comparisons to figure out which files should be skipped or copied. > Since different cloud stores have different checksum algorithms we should > check for modification time as well to the checks. > This would ensure that while performing -update if the files are perceived to > be out of sync we should copy them. The machines between which the file > transfers occur should be in time sync to avoid any extra copies. > Improving testing and documentation for modification time checks between > different object stores to ensure no incorrect skipping of files. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org