[
https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935743#comment-15935743
]
Omkar Aradhya K S commented on HADOOP-11794:
--------------------------------------------
{quote}
The main reason of checking DistributedFileSystem is the support of
getBlockLocations, and concat feature. I'm not sure whether we can assume other
File System support that.
{quote}
The *getFileBlockLocations* and *concat* are APIs that are part of
*FileSystem.java* from [hadoop
v1.2.1|https://hadoop.apache.org/docs/r1.2.1/api/index.html]
{quote}
The current patch is for trunk where client and server code are separated. When
we backport this change to other version of hadoop, we can make the change
accordingly, for example, to use DFSUtil.
{quote}
You could just use the default constructor that would internally get the
NNAddress:
{code}
final DFSClient dfs = new DFSClient(conf);
{code}
> distcp can copy blocks in parallel
> ----------------------------------
>
> Key: HADOOP-11794
> URL: https://issues.apache.org/jira/browse/HADOOP-11794
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
> Affects Versions: 0.21.0
> Reporter: dhruba borthakur
> Assignee: Yongjun Zhang
> Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch,
> HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch,
> HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch,
> MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are
> greater than 1 TB with a block size of 1 GB. If we use distcp to copy these
> files, the tasks either take a long long long time or finally fails. A better
> way for distcp would be to copy all the source blocks in parallel, and then
> stich the blocks back to files at the destination via the HDFS Concat API
> (HDFS-222)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
