[
https://issues.apache.org/jira/browse/MAPREDUCE-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105208#comment-13105208
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2998:
----------------------------------------------------
>From both the logs you pasted, the job clearly failed, you need to debug why
>by looking at AM logs.
The pasted exceptions are happening because MR AM exited when some call from
client was in progress. This is expected in some cases. The retry should
eventually figure out the final state of the job from RM and subsequently form
JobHistoryServer. BTW, do you have JobHistoryServer up?
Side note: Reading the logs pasted on JIRA is so hard, can you please attach
the full JobClient as a separate file the next time you run into this? Thanks!
> Failing to contact Am/History for jobs: java.io.EOFException in
> DataInputStream
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2998
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2998
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Jeffrey Naisbitt
>
> I am getting an exception frequently when running my jobs on a single-node
> cluster. It happens with basically any job I run: sometimes the job will
> work, but most of the time I get this exception (in this case, I was running
> a simple wordcount from the examples jar - where I got the exception 4 times
> in a row, and then the job worked the fifth time I submitted it).
> Sometimes restarting the namenode, resourcemanager, and historyserver helps -
> but not always. Several other developers have seen this problem.
> 11/09/12 17:17:50 INFO mapred.YARNRunner: AppMaster capability = memory:
> 2048,
> 11/09/12 17:17:51 INFO mapred.YARNRunner: Command to launch container for
> ApplicationMaster is : $JAVA_HOME/bin/java -Dhadoop.root.logger=DEBUG,console
> -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1315847180566 6
> <FAILCOUNT> 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
> 11/09/12 17:17:51 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1315847180566_6 to ResourceManager
> 11/09/12 17:17:51 INFO mapred.ClientCache: Connecting to HistoryServer at:
> 0.0.0.0:10020
> 11/09/12 17:17:51 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/12 17:17:51 INFO mapred.ClientCache: Connected to HistoryServer at:
> 0.0.0.0:10020
> 11/09/12 17:17:51 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol
> 11/09/12 17:17:51 INFO mapreduce.Job: Running job: job_1315847180566_0006
> 11/09/12 17:17:52 INFO mapreduce.Job: map 0% reduce 0%
> 11/09/12 17:18:00 INFO mapred.ClientServiceDelegate: Tracking Url of JOB is
> <IP-ADDRESS>:55361
> 11/09/12 17:18:00 INFO mapred.ClientServiceDelegate: Connecting to
> <IP-ADDRESS>:43465
> 11/09/12 17:18:00 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/12 17:18:00 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol
> 11/09/12 17:18:01 INFO mapred.ClientServiceDelegate: Failed to contact
> AM/History for job job_1315847180566_0006 Will retry..
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBClientImpl.java:179)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:237)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:276)
> at
> org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:547)
> at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:540)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1144)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1092)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
> Caused by: com.google.protobuf.ServiceException: java.io.IOException: Call to
> /<IP-ADDRESS>:43465 failed on local exception: java.io.EOFException
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139)
> at $Proxy8.getTaskAttemptCompletionEvents(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBClientImpl.java:172)
> ... 23 more
> Caused by: java.io.IOException: Call to /<IP-ADDRESS>:43465 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1119)
> at org.apache.hadoop.ipc.Client.call(Client.java:1087)
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136)
> ... 25 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:816)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:754)
> 11/09/12 17:18:01 INFO mapreduce.Job: Job job_1315847180566_0006 failed with
> state FAILED
> 11/09/12 17:18:01 INFO mapreduce.Job: Counters: 0
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
