Hi, I have set up a single node Hadoop 2.2.0 . I can submit wordcount hadoop job to YARN from the command line. It run successfully and I checked the result is in hdfs. matmsh@gauss:~> hadoop jar ~/installed/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount input output6 14/06/28 13:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/06/28 13:40:00 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/06/28 13:40:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 14/06/28 13:40:00 INFO input.FileInputFormat: Total input paths to process : 1 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: number of splits:1 14/06/28 13:40:00 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name 14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 14/06/28 13:40:00 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 14/06/28 13:40:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local28288787_0001 14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/06/28 13:40:00 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 14/06/28 13:40:00 INFO mapreduce.Job: Running job: job_local28288787_0001 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter set in config null 14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Waiting for map tasks 14/06/28 13:40:01 INFO mapred.LocalJobRunner: Starting task: attempt_local28288787_0001_m_000000_0 14/06/28 13:40:01 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 14/06/28 13:40:01 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/matmsh/input/getdoc.txt:0+23992 14/06/28 13:40:01 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 14/06/28 13:40:01 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) .....
However, the above job does not show up at the web UI : http://localhost:8088/cluster/apps My configurations are below, mapred-site.xml : <configuration> <property> <value>mapreduce.framework.name</value> <name>yarn</name> </property> </configuration> ------- hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/matmsh/temp/hadoop2/namenode</value> <description>The name of the default file system.</description> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/home/matmsh/temp/hadoop2/datanode</value> <description>The name of the default file system.</description> </property> </configuration> ---------------- yarn-site.xml <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> </configuration> ------------------------------------ What other configurations are needed to show the job in the web UI ? Thanks in advance for any assistance ! Shing
