Hi I am facing an error during Java Map reduce code execution in using Hadoop 2.2 The FinalApplicationStatus is successful. I can find the output in desired location but It throws error while redirecting to job history server (please refer to red line in the logs below).
*Explanation in detail:* Input File to the Java MR job: 100 files ~ each with 300 MB size , Total size = 34 GB *Case 1: (1 Job ) Error* Run one MR job for all 100 files as input to the job ( Run time - 1 hour) It throws error as given in job logs - marked in red color below. *Case 2: (100 job)Successful* Run one MR job for each 100 files as input to the job ( Run time - 40 sec per job) I can see all 100 output and all job ran successfully.Also for all these jobs it successfully redirected to job history server and displayed all counter in terminal. (No error was seen in this case) After Case 2, I cannot blame to my cluster configuration. *How do I fix this ? Please help.* I could also find few unresolved JIRA tickets and discussion for similar case(links below): 1) Similar case http://mail-archives.apache.org/mod_mbox/oozie-user/201402.mbox/%3CCAAu13zFQWQuVo-ShJYVT-o+D=SC6-4Zhwn8UKPsB31wS62=0...@mail.gmail.com%3E 2) Exact Same Case http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201311.mbox/%3CCANXCz3RC5aCgeAMro=wS6TbwjsQVx9kemJd28kpmjMTp5hdX=q...@mail.gmail.com%3E 3) Similar Case http://stackoverflow.com/questions/17930644/connection-error-in-apache-pig *Job Logs:* 14/07/22 06:37:11 INFO mapreduce.Job: map 100% reduce 98% 14/07/22 06:37:44 INFO mapreduce.Job: map 100% reduce 99% 14/07/22 06:38:30 INFO mapreduce.Job: map 100% reduce 100% 14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 14/07/22 06:39:02 ERROR security.UserGroupInformation: PriviledgedActionException as: rohitsarewar (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:269) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047) Exception in thread "main" java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:269) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047) at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:330) at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:382) at org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:529) at org.apache.hadoop.mapreduce.Job$5.run(Job.java:668) at org.apache.hadoop.mapreduce.Job$5.run(Job.java:665) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:665) at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1349) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289) at com.bigdata.mapreduce.esc.escDriver.main(escDriver.java:23) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): j at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolH at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBSer at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2. at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(P at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine at com.sun.proxy.$Proxy12.getTaskAttemptCompletionEvents(Unknown Source) at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClie at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDeleg ... 16 more Regards Rohit Sarewar
