[
https://issues.apache.org/jira/browse/HADOOP-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590853#action_12590853
]
Vinod Kumar Vavilapalli commented on HADOOP-3276:
-------------------------------------------------
This particular problem occurs because of improper setup of Mapred system
directory in DFS. But, this can be hit in general when either user submits jobs
or HOD continues running script commands, even while JT is still in the state
of initialization due to various reasons. HOD currently puts in its best effort
to avoid this - ping Mapred Jetty if it is alive - the last thing that can be
done to check JT status by HOD from outside without any support in Hadoop. And
as the above situation makes it clear, we definitely need support from Hadoop
to know when JT can really accepts jobs. Filed a Jira (HADOOP-3289) to address
this.
> Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running
> hadoop job on cluster allocated by hod.
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3276
> URL: https://issues.apache.org/jira/browse/HADOOP-3276
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.17.0
> Reporter: Karam Singh
> Priority: Minor
>
> Allocated cluster using hod of 5 nodes . It returned the both Mapred and Hdfs
> UI addresses. Also checked out of nodes on 3 taskterakers nodes were using
> getTaskTrackers() api.
> Then tried to run wordcount example and got following -:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.mapred.JobTracker$IllegalStateException: Job tracker still
> initializing
> at
> org.apache.hadoop.mapred.JobTracker.ensureRunning(JobTracker.java:1722)
> at
> org.apache.hadoop.mapred.JobTracker.getNewJobId(JobTracker.java:1730)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at $Proxy1.getNewJobId(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.getNewJobId(Unknown Source)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:696)
> (Here dfs was static with permissions on.)
> On further checking there was problem in static dfs setting, but Web UI was
> getting started resulting successful allocation, but was not able create
> mparedsystem directory causing JobTracker$IllegalStateException.
> So, It seems if Web UI up then hod assumes that Mapred JT is running
> (This is rare condition)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
