[
https://issues.apache.org/jira/browse/HADOOP-16756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992581#comment-16992581
]
Daisuke Kobayashi edited comment on HADOOP-16756 at 12/10/19 2:05 PM:
----------------------------------------------------------------------
[[email protected]], hmm really. Here's my command which is pretty simple:
{noformat}
hadoop distcp -Dfs.s3a.access.key=xxx -Dfs.s3a.secret.key=xxx -update
-skipcrccheck /user/root/tmp/ s3a://<my-bucket>/tmp/
{noformat}
The version I've tested was CDH 6.3.2.
was (Author: daisuke.kobayashi):
[[email protected]], hmm really. Here's my command which is pretty simple:
{noformat}
hadoop distcp -Dfs.s3a.access.key=xxx -Dfs.s3a.secret.key=xxx -update
-skipcrccheck /user/root/tmp/ s3a://<my-bucket>/tmp/
{noformat}
> Inconsistent Behavior on distcp -update over S3
> -----------------------------------------------
>
> Key: HADOOP-16756
> URL: https://issues.apache.org/jira/browse/HADOOP-16756
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, tools/distcp
> Affects Versions: 3.3.0
> Reporter: Daisuke Kobayashi
> Priority: Major
>
> Distcp over S3A always copies all source files no matter the files are
> changed or not. This is opposite to the statement in the doc below.
> [http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html]
> {noformat}
> And to use -update to only copy changed files.
> {noformat}
> CopyMapper compares file length as well as block size before copying. While
> the file length should match, the block size does not. This is apparently
> because the returned block size from S3A is always 32MB.
> [https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L348]
> I'd suppose we should update the documentation or make code change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
