Looks like your cluster has Kerberos enabled and your job wasn't able to authenticate with the RM at startup On Oct 29, 2015 10:18 AM, "Nicola Barbieri" <[email protected]> wrote:
> Is there anybody that can help me in understanding this exception? > It could be related to an heavy workload, but unfortunately there aren't > much details in the stack trace. > > Thanks, > Nicola > > > 2015-10-29 17:05:30,145 FATAL [org.apache.giraph.master.MasterThread] > org.apache.giraph.master.BspServiceMaster: failJob: exception > java.lang.IllegalStateException: ******* WORKERS [Worker(hostname= > gsta32845.tan.ygrid.yahoo.com hostOrIp=gsta32845.tan.ygrid.yahoo.com, > MRtaskID=36, port=30036), Worker(hostname=gsta33083.tan.ygrid.yahoo.com > hostOrIp=gsta33083.tan.ygrid.yahoo.com, MRtaskID=90, port=30090)] FAILED > ******* > 2015-10-29 17:05:30,295 INFO [org.apache.giraph.master.MasterThread] > org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at > nodename.com/98.138.174.71:8032 > 2015-10-29 17:05:30,666 WARN [org.apache.giraph.master.MasterThread] > org.apache.hadoop.ipc.Client: Exception encountered while connecting to the > server : org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > 2015-10-29 17:05:30,774 WARN [org.apache.giraph.master.MasterThread] > org.apache.hadoop.ipc.Client: Exception encountered while connecting to the > server : org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > 2015-10-29 17:05:30,877 WARN [org.apache.giraph.master.MasterThread] > org.apache.hadoop.ipc.Client: Exception encountered while connecting to the > server : org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > 2015-10-29 17:05:30,980 ERROR [org.apache.giraph.master.MasterThread] > org.apache.giraph.master.MasterThread: masterThread: Master algorithm > failed with RuntimeException > java.lang.RuntimeException: java.io.IOException: Failed on local > exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: " > nodename.com/xx.xxx.xxx.xx"; destination host is: "nodename.com":8032; > at > org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:382) > at > org.apache.giraph.master.BspServiceMaster.setJobStateFailed(BspServiceMaster.java:311) > at > org.apache.giraph.master.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1358) > at > org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1592) > at org.apache.giraph.master.MasterThread.run(MasterThread.java:124) > Caused by: java.io.IOException: Failed on local exception: > java.io.IOException: org.apache.hadoop.security.AccessControlException: > Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host > is: "nodename.com/xx.xxx.xxx.xx"; destination host is: "nodename.com > ":8032; > at > org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:357) > at > org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:428) > at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:575) > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:578) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694) > at > org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:578) > at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:596) > at > org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:374) > ... 4 more > 2015-10-29 17:05:30,982 FATAL [org.apache.giraph.master.MasterThread] > org.apache.giraph.graph.GraphTaskManager: uncaughtException: > OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread, > msg = java.lang.RuntimeException: java.io.IOException: Failed on local > exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: " > nodename.com/xx.xxx.xxx.xx"; destination host is: "nodename.com":8032; , > exiting... > java.lang.IllegalStateException: java.lang.RuntimeException: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: " > nodename.com/xx.xxx.xxx.xx"; destination host is: "nodename.com":8032; > at org.apache.giraph.master.MasterThread.run(MasterThread.java:194) > Caused by: java.lang.RuntimeException: java.io.IOException: Failed on > local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: " > nodename.com/xx.xxx.xxx.xx"; destination host is: "nodename.com":8032; > at > org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:382) > at > org.apache.giraph.master.BspServiceMaster.setJobStateFailed(BspServiceMaster.java:311) > at > org.apache.giraph.master.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1358) > at > org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1592) > at org.apache.giraph.master.MasterThread.run(MasterThread.java:124) > Caused by: java.io.IOException: Failed on local exception: > java.io.IOException: org.apache.hadoop.security.AccessControlException: > Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host > is: "nodename.com/xx.xxx.xxx.xx"; destination host is: "nodename.com > ":8032; > at > org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:357) > at > org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:428) > at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:575) > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:578) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694) > at > org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:578) > at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:596) > at > org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:374) > ... 4 more >
