[
https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937523#comment-15937523
]
Yongjun Zhang commented on HADOOP-11794:
----------------------------------------
HI [~chris.douglas],
Sorry did not see your last post before I did my previous comment and I had to
be offline for some time.
In a case that source support getBlockLocations and target doesn't support
concat, the current patch would split a file and copy them into chunk files at
target, then at commit stage we will find concat doesn't work, thus the target
is polluted (inconsistent state). At this point, distcp may have been running
for very long time. To remedy that, I will add a "concat" check at the same
place where I check DistributedFileSystem and catch
UnsupportedOperationException (as [[email protected]] suggested). This will
address the last item in my previous comment.
Thanks.
> distcp can copy blocks in parallel
> ----------------------------------
>
> Key: HADOOP-11794
> URL: https://issues.apache.org/jira/browse/HADOOP-11794
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
> Affects Versions: 0.21.0
> Reporter: dhruba borthakur
> Assignee: Yongjun Zhang
> Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch,
> HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch,
> HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch,
> MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are
> greater than 1 TB with a block size of 1 GB. If we use distcp to copy these
> files, the tasks either take a long long long time or finally fails. A better
> way for distcp would be to copy all the source blocks in parallel, and then
> stich the blocks back to files at the destination via the HDFS Concat API
> (HDFS-222)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
