[
https://issues.apache.org/jira/browse/HDFS-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-2392:
------------------------------
Attachment: HDFS-2392.patch
Fix to call setDelegationToken with a token located in the cache.
Move token initialization into a new method. Update initialize method to call
the token init.
Using aforementioned change to write unit tests to verify that the token is
set, and that the correct token is set.
> Dist with hftp is failing again
> -------------------------------
>
> Key: HDFS-2392
> URL: https://issues.apache.org/jira/browse/HDFS-2392
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.205.0
> Reporter: Rajit Saha
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HDFS-2392.patch
>
>
> $ hadoop distcp hftp://<NN Hostname>:50070/user/hadoopqa/input1/part-00000
> /user/hadoopqa/out3
> 11/09/30 18:57:59 INFO tools.DistCp: srcPaths=[hftp://<NN
> Hostname>:50070/user/hadoopqa/input1/part-00000]
> 11/09/30 18:57:59 INFO tools.DistCp: destPath=/user/hadoopqa/out3
> 11/09/30 18:58:00 INFO security.TokenCache: Got dt for
> hftp://<NN Hostname>:50070/user/hadoopqa/input1/part-00000;uri=<NN
> IP>:50470;t.service=<NN IP>:50470
> 11/09/30 18:58:00 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 24
> for hadoopqa on <NN IP>:8020
> 11/09/30 18:58:00 INFO security.TokenCache: Got dt for
> /user/hadoopqa/out3;uri=<NN IP>:8020;t.service=<NN IP>:8020
> 11/09/30 18:58:00 INFO tools.DistCp: /user/hadoopqa/out3 does not exist.
> 11/09/30 18:58:00 INFO tools.DistCp: sourcePathsCount=1
> 11/09/30 18:58:00 INFO tools.DistCp: filesToCopyCount=1
> 11/09/30 18:58:00 INFO tools.DistCp: bytesToCopyCount=1.0g
> 11/09/30 18:58:01 INFO mapred.JobClient: Running job: job_201109300819_0007
> 11/09/30 18:58:02 INFO mapred.JobClient: map 0% reduce 0%
> 11/09/30 18:58:25 INFO mapred.JobClient: Task Id :
> attempt_201109300819_0007_m_000000_0, Status : FAILED
> java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
> at
> org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> 11/09/30 18:58:41 INFO mapred.JobClient: Task Id :
> attempt_201109300819_0007_m_000000_1, Status : FAILED
> java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
> at
> org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> 11/09/30 18:58:56 INFO mapred.JobClient: Task Id :
> attempt_201109300819_0007_m_000000_2, Status : FAILED
> java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
> at
> org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> 11/09/30 18:59:14 INFO mapred.JobClient: Job complete: job_201109300819_0007
> 11/09/30 18:59:14 INFO mapred.JobClient: Counters: 6
> 11/09/30 18:59:14 INFO mapred.JobClient: Job Counters
> 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=62380
> 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all reduces
> waiting after reserving slots (ms)=0
> 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 11/09/30 18:59:14 INFO mapred.JobClient: Launched map tasks=4
> 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 11/09/30 18:59:14 INFO mapred.JobClient: Failed map tasks=1
> 11/09/30 18:59:14 INFO mapred.JobClient: Job Failed: # of failed Map Tasks
> exceeded allowed limit. FailedCount: 1.
> LastFailedTask: task_201109300819_0007_m_000000
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1257)
> at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
