[
https://issues.apache.org/jira/browse/MAPREDUCE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907446#comment-14907446
]
Harsh J commented on MAPREDUCE-6471:
------------------------------------
The changes in the options section look good to me, but the note added in the
{{-update and -overwrite}} behaviour got me reading it entirely, and somehow
the last example block looks incorrect. Its not directly related to your
change, but there's a few problems in the below, unless am wrong:
{code}
Now, consider the following copy operation:
distcp hdfs://nn1:8020/source/first hdfs://nn1:8020/source/second
hdfs://nn2:8020/target
With sources/sizes:
hdfs://nn1:8020/source/first/1 32
hdfs://nn1:8020/source/first/2 32
hdfs://nn1:8020/source/second/10 64
hdfs://nn1:8020/source/second/20 32
And destination/sizes:
hdfs://nn2:8020/target/1 32
hdfs://nn2:8020/target/10 32
hdfs://nn2:8020/target/20 64
Will effect:
hdfs://nn2:8020/target/1 32
hdfs://nn2:8020/target/2 32
hdfs://nn2:8020/target/10 64
hdfs://nn2:8020/target/20 32
1 is skipped because the file-length and contents match. 2 is copied because it
doesn’t exist at the target. 10 and 20 are overwritten since the contents don’t
match the source.
If -update is used, 1 is overwritten as well.
{code}
Those last two lines, I think should read instead as:
{code}
If `-update` is used, 1 is skipped because the file-length and contents match.
2 is copied because it doesn’t exist at the target. 10 and 20 are overwritten
since the contents don’t match the source.
If `-overwrite` is used, 1 is overwritten as well.
{code}
Or with the {{-append}} change added:
{code}
If `-update` is used, 1 is skipped because the file-length and contents match.
2 is copied because it doesn’t exist at the target. 10 and 20 are overwritten
since the contents don’t match the source. However, if `-append` is
additionally used, then only 10 is overwritten (source length less than
destination) and 20 is appended with the change in file (if the files match up
to the destination's original length).
If `-overwrite` is used, 1 is overwritten as well.
{code}
Thoughts?
> Document distcp incremental copy
> ---------------------------------
>
> Key: MAPREDUCE-6471
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6471
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 2.7.1
> Reporter: Arpit Agarwal
> Assignee: Neelesh Srinivas Salian
> Labels: newbie
> Attachments: MAPREDUCE-6471.001.patch, MAPREDUCE-6471.002.patch
>
>
> MAPREDUCE-5899 added distcp support for incremental copy with a new
> {{append}} flag.
> It should be documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
