Vinod, I am confused here. So could you please explain what actually happened under the hood if " mapreduce.framework.name" is set to "classic" on the cluster side? Or it is supposed to be set to "yarn" in the first place?
Thanks. Kim On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli <[email protected] > wrote: > You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported > nor even possible. > > +Vinod > > On Apr 16, 2014, at 2:27 PM, Kim Chew <[email protected]> wrote: > > I have a cluster running Hadoop 2 but it is not running YARN, i.e. " > mapreduce.framework.name" is set to "classic" therefore the > ResourceManager is not running. > > On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to > the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like, > > <property> > <!-- Pointed to the remote JobTracker --> > <name>mapred.job.tracker</name> > <value>172.31.3.150:8021</value> > </property> > > Not surprisingly I got a version mismatched when I submit my job using the > Hadoop-1.1.1 jars, > > org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot > communicate with client version 4 > at org.apache.hadoop.ipc.Client.call(Client.java:1107) > > So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 > jars. Here is how my Hadoop 2 mapred-site.xml looks like, > > <property> > <!-- Pointed to the remote JobTracker --> > <name>mapreduce.job.tracker.address</name> > <value>172.31.3.150:8021</value> > </property> > <property> > <name>mapreduce.framework.name</name> > <value>yarn</value> > </property> > > Note that I have to set "mapreduce.framework.name" to "yarn" otherwise > the job will be run locally instead of on the targeted cluster. But my > targeted cluster is not running YARN as stated above, > > 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at / > 172.31.3.150:8032 > 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: > hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); > retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, > sleepTime=1 SECONDS) > > (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in > yarn-site.xml on my client.) > > Therefore it seems to me that it does not matter I have to recompile my > job with Hadoop 2 or not. The question is what should I do to enable > submitting my job remotely to the Hadoop 2 cluster ? What are the > configurations I need to set on the client side? > > The only solution I can think of is to enable YARN on the Hadoop 2 cluster > but is it necessary? > > I am running out of pointers and stuck 8-( > > TIA > > Kim > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
