[ https://issues.apache.org/jira/browse/MAPREDUCE-5589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798169#comment-13798169 ]
Benoy Antony commented on MAPREDUCE-5589: ----------------------------------------- Proper error output by JobLocalizer is displayed when the patch is applied. Error initializing attempt_201310140952_0001_m_000002_0: java.io.IOException: Job initialization failed (255) with output: Reading task controller config from /etc/hadoop/taskcontroller.cfg main : command provided 0 main : user is bantony Good mapred-local-dirs are /hadoop12/scratch,/hadoop04/scratch,/hadoop09/scratch,/hadoop05/scratch,/hadoop01/scratch,/hadoop11/scratch,/hadoop10/scratch,/hadoop08/scratch,/hadoop06/scratch,/hadoop07/scratch,/hadoop02/scratch java.io.IOException: Call to localhost/127.0.0.1:60487 failed on local exception: java.io.IOException: javax.security.sasl.SaslException: DIGEST-MD5: No common protection layer between client and server at org.apache.hadoop.ipc.Client.wrapException(Client.java:1155) at org.apache.hadoop.ipc.Client.call(Client.java:1123) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy5.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:414) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:392) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:374) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:444) at org.apache.hadoop.mapred.JobLocalizer$2.run(JobLocalizer.java:529) at org.apache.hadoop.mapred.JobLocalizer$2.run(JobLocalizer.java:527) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.JobLocalizer.main(JobLocalizer.java:526) Caused by: java.io.IOException: javax.security.sasl.SaslException: DIGEST-MD5: No common protection layer between client and server at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:568) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:513) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:616) at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:203) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1254) at org.apache.hadoop.ipc.Client.call(Client.java:1098) ... 13 more Caused by: javax.security.sasl.SaslException: DIGEST-MD5: No common protection layer between client and server at com.sun.security.sasl.digest.DigestMD5Client.checkQopSupport(DigestMD5Client.java:396) at com.sun.security.sasl.digest.DigestMD5Client.evaluateChallenge(DigestMD5Client.java:208) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:168) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:410) at org.apache.hadoop.ipc.Client$Connection.access$1300(Client.java:203) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:609) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:606) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:605) ... 16 more at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193) at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1340) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1315) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1230) at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2641) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:255) at org.apache.hadoop.util.Shell.run(Shell.java:182) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186) ... 8 more This is not a problem in Yarn. > MapReduce Job setup error leaves no useful info to users (when > LinuxTaskController is used) > -------------------------------------------------------------------------------------------- > > Key: MAPREDUCE-5589 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5589 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 > Affects Versions: 1.2.1 > Environment: I found that a significant portion of the main() in > JobLocalizer will not have its output displayed in the tasklogs. > Before fix : > Error initializing attempt_201310122204_0002_m_000002_0: > java.io.IOException: Job initialization failed (1) with output: Reading task > controller config from /etc/hadoop/taskcontroller.cfg > main : command provided 0 > main : user is bantony > Good mapred-local-dirs are > /hadoop12/scratch,/hadoop04/scratch,/hadoop09/scratch,/hadoop03/scratch,/hadoop05/scratch,/hadoop01/scratch,/hadoop11/scratch,/hadoop10/scratch,/hadoop08/scratch,/hadoop06/scratch,/hadoop07/scratch,/hadoop02/scratch > at > org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1340) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) > at > org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1315) > at > org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1230) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2641) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.hadoop.util.Shell$ExitCodeException: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:255) > at org.apache.hadoop.util.Shell.run(Shell.java:182) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) > at > org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186) > ... 8 more > Reporter: Benoy Antony > Assignee: Benoy Antony > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5368.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)