[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678178#comment-17678178
]
ASF GitHub Bot commented on HADOOP-18596:
-----------------------------------------
mehakmeet opened a new pull request, #5308:
URL: https://github.com/apache/hadoop/pull/5308
### Description of PR
Using modification time as a way to add more checks to determine if distcp
-update should skip a file or not.
In specific cases like the same file name, and size but different content we
used to incorrectly skip files in update since there is no checksum comparison
between object stores with different algorithm for it, to mitigate this we
introduce comparing modification time between the target file and the source.
### How was this patch tested?
Manually tested on an environment after reproducing the scenario where we
might incorrectly skip a file.
Added a test in `AbstractContractDistCpTest.java` to test by changing the
target file's modification time to emulate the scenario.
Tested on S3A(ap-south-1), ABFS(us-west-2), and LocalFS, and the test was
successful.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [X] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Distcp -update between different cloud stores to use modification time while
> checking for file skip.
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18596
> URL: https://issues.apache.org/jira/browse/HADOOP-18596
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
>
> Distcp -update currently relies on File size, block size, and Checksum
> comparisons to figure out which files should be skipped or copied.
> Since different cloud stores have different checksum algorithms we should
> check for modification time as well to the checks.
> This would ensure that while performing -update if the files are perceived to
> be out of sync we should copy them. The machines between which the file
> transfers occur should be in time sync to avoid any extra copies.
> Improving testing and documentation for modification time checks between
> different object stores to ensure no incorrect skipping of files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
