[
https://issues.apache.org/jira/browse/HDFS-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852524#comment-13852524
]
Yongjun Zhang commented on HDFS-5685:
-------------------------------------
I've identified that the problem is in the implementation of DistCp, not
because of MR. It removes file prematurely
and then try to remove again. Will provide patch soon.
It shows up in DistCp implemented with MR1, and I'm about the examine the
DistCp implementation with MR2.
> DistCp will fail to copy with -delete switch
> --------------------------------------------
>
> Key: HDFS-5685
> URL: https://issues.apache.org/jira/browse/HDFS-5685
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 1.0.0
> Reporter: Yongjun Zhang
> Assignee: Yongjun Zhang
>
> When using distcp command to copy files with -delete switch, running as user
> <xyz>,
> hadoop distcp -p -i -update -delete hdfs://srchost:<port>/user
> hdfs://dsthost:<port>/user
> It fails with the following exception:
> Copy failed: java.io.FileNotFoundException: File does not exist:
> hdfs://dsthost:<port>/user/xyz/.stagingdistcp_urjb0g/_distcp_src_files
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:557)
> at
> org.apache.hadoop.tools.DistCp$CopyInputFormat.getSplits(DistCp.java:266)
> at
> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1081)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1073)
> at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
> at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
