Hi Marcus, Sorry i forgot to mention that Job history server is installed and running and AFAIK resourcemanager is responsible for running MR jobs. Historyserver is only used to get info about MR jobs.
Thanks, Anil On Wed, Jun 13, 2012 at 2:04 PM, Marcos Ortiz <mlor...@uci.cu> wrote: > According to the CDH 4 official documentation, you should install a > JobHistory server for your MRv2 (YARN) > cluster. > https://ccp.cloudera.com/**display/CDH4DOC/Deploying+** > MapReduce+v2+%28YARN%29+on+a+**Cluster<https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster> > > How to configure the HistoryServer > https://ccp.cloudera.com/**display/CDH4DOC/Deploying+** > MapReduce+v2+%28YARN%29+on+a+**Cluster#DeployingMapReducev2%** > 28YARN%29onaCluster-Step3<https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster#DeployingMapReducev2%28YARN%29onaCluster-Step3> > > > > > On 06/13/2012 03:16 PM, anil gupta wrote: > >> Hi All >> >> I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5 >> nodes in my cluster(2 Admin Node and 3 DN). >> My resourcemanager is up and running and showing that all three DN are >> running the nodemanager. HDFS is also working fine and showing 3 DN's. >> >> But when i fire the pi example job. It starts to run in Local mode. >> Here is the console output: >> sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/**hadoop-mapreduce- >> examples.jar pi 10 1000000000 >> Number of Maps = 10 >> Samples per Map = 1000000000 >> Wrote input for Map #0 >> Wrote input for Map #1 >> Wrote input for Map #2 >> Wrote input for Map #3 >> Wrote input for Map #4 >> Wrote input for Map #5 >> Wrote input for Map #6 >> Wrote input for Map #7 >> Wrote input for Map #8 >> Wrote input for Map #9 >> Starting Job >> 12/06/13 12:03:27 WARN conf.Configuration: session.id is deprecated. >> Instead, use dfs.metrics.session-id >> 12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM Metrics with >> processName=JobTracker, sessionId= >> 12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the native-hadoop >> library >> 12/06/13 12:03:27 WARN mapred.JobClient: Use GenericOptionsParser for >> parsing the arguments. Applications should implement Tool for the >> same. >> 12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input paths to >> process : 10 >> 12/06/13 12:03:29 INFO mapred.JobClient: Running job: job_local_0001 >> 12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter set in >> config null >> 12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is >> org.apache.hadoop.mapred.**FileOutputCommitter >> 12/06/13 12:03:29 WARN mapreduce.Counters: Group >> org.apache.hadoop.mapred.Task$**Counter is deprecated. Use >> org.apache.hadoop.mapreduce.**TaskCounter instead >> 12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with exit code >> 0 >> 12/06/13 12:03:29 INFO mapred.Task: Using ResourceCalculatorPlugin : >> org.apache.hadoop.util.**LinuxResourceCalculatorPlugin@**3d46e381 >> 12/06/13 12:03:29 WARN mapreduce.Counters: Counter name >> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group >> name and BYTES_READ as counter name instead >> 12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1 >> 12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100 >> 12/06/13 12:03:30 INFO mapred.MapTask: data buffer = 79691776/99614720 >> 12/06/13 12:03:30 INFO mapred.MapTask: record buffer = 262144/327680 >> 12/06/13 12:03:30 INFO mapred.JobClient: map 0% reduce 0% >> 12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000 >> samples. >> 12/06/13 12:03:36 INFO mapred.JobClient: map 100% reduce 0% >> 12/06/13 12:03:38 INFO mapred.LocalJobRunner: Generated 151872000 >> samples. >> >> Here is the content of yarn-site.xml: >> >> <configuration> >> <property> >> <name>yarn.nodemanager.aux-**services</name> >> <value>mapreduce.shuffle</**value> >> </property> >> >> <property> >> <name>yarn.nodemanager.aux-**services.mapreduce.shuffle.** >> class</name> >> <value>org.apache.hadoop.**mapred.ShuffleHandler</value> >> </property> >> >> <property> >> <name>yarn.log-aggregation-**enable</name> >> <value>true</value> >> </property> >> >> <property> >> <description>List of directories to store localized files in.</ >> description> >> <name>yarn.nodemanager.local-**dirs</name> >> <value>/disk/yarn/local</**value> >> </property> >> >> <property> >> <description>Where to store container logs.</description> >> <name>yarn.nodemanager.log-**dirs</name> >> <value>/disk/yarn/logs</value> >> </property> >> >> <property> >> <description>Where to aggregate logs to.</description> >> <name>yarn.nodemanager.remote-**app-log-dir</name> >> <value>/var/log/hadoop-yarn/**apps</value> >> </property> >> >> <property> >> <description>Classpath for typical applications.</description> >> <name>yarn.application.**classpath</name> >> <value> >> $HADOOP_CONF_DIR, >> $HADOOP_COMMON_HOME/*,$HADOOP_**COMMON_HOME/lib/*, >> $HADOOP_HDFS_HOME/*,$HADOOP_**HDFS_HOME/lib/*, >> $HADOOP_MAPRED_HOME/*,$HADOOP_**MAPRED_HOME/lib/*, >> $YARN_HOME/*,$YARN_HOME/lib/* >> </value> >> </property> >> <property> >> <name>yarn.resourcemanager.**resource-tracker.address</**name> >> <value>ihub-an-g1:8025</value> >> </property> >> <property> >> <name>yarn.resourcemanager.**address</name> >> <value>ihub-an-g1:8040</value> >> </property> >> <property> >> <name>yarn.resourcemanager.**scheduler.address</name> >> <value>ihub-an-g1:8030</value> >> </property> >> <property> >> <name>yarn.resourcemanager.**admin.address</name> >> <value>ihub-an-g1:8141</value> >> </property> >> <property> >> <name>yarn.resourcemanager.**webapp.address</name> >> <value>ihub-an-g1:8088</value> >> </property> >> <property> >> <name>mapreduce.jobhistory.**intermediate-done-dir</name> >> <value>/disk/mapred/**jobhistory/intermediate/done</**value> >> </property> >> <property> >> <name>>mapreduce.jobhistory.**done-dir</name> >> <value>/disk/mapred/**jobhistory/done</value> >> </property> >> </configuration> >> >> Can anyone tell me what is the problem over here? Appreciate your >> help. >> Thanks, >> Anil Gupta >> >> >> >> > -- > Marcos Luis OrtÃz Valmaseda > Data Engineer&& Sr. System Administrator at UCI > http://marcosluis2186.**posterous.com<http://marcosluis2186.posterous.com> > > http://www.linkedin.com/in/**marcosluis2186<http://www.linkedin.com/in/marcosluis2186> > Twitter: @marcosluis2186 > > > 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS > INFORMATICAS... > CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION > > http://www.uci.cu > http://www.facebook.com/**universidad.uci<http://www.facebook.com/universidad.uci> > http://www.flickr.com/photos/**universidad_uci<http://www.flickr.com/photos/universidad_uci> > -- Thanks & Regards, Anil Gupta
