Answers below.
From: Omkar Joshi [mailto:[email protected]] Sent: Friday, April 26, 2013 7:15 PM To: [email protected] Subject: Re: M/R Staticstics Have you enabled security? No can you share the output for your hdfs? bin/hadoop fs -ls / kevin@devUbuntu05:~$ hadoop fs -ls / Found 2 items drwxrwxrwx - hdfs supergroup 0 2013-04-26 13:33 /tmp drwxr-xr-x - hdfs supergroup 0 2013-04-19 16:40 /user and is /tmp/hadoop-yarn/staging/history/done directory present in hdfs ? if so then what permissions? kevin@devUbuntu05:~$ hadoop fs -ls -R /tmp drwxrwx--- - mapred supergroup 0 2013-04-26 13:33 /tmp/hadoop-yarn ls: Permission denied: user=kevin, access=READ_EXECUTE, inode="/tmp/hadoop-yarn":mapred:supergroup:drwxrwx--- kevin@devUbuntu05:~$ sudo -u hdfs hadoop fs -ls -R /tmp [sudo] password for kevin: drwxrwx--- - mapred supergroup 0 2013-04-26 13:33 /tmp/hadoop-yarn drwxrwx--- - mapred supergroup 0 2013-04-26 13:33 /tmp/hadoop-yarn/staging drwxrwx--- - mapred supergroup 0 2013-04-26 13:33 /tmp/hadoop-yarn/staging/history drwxrwx--- - mapred supergroup 0 2013-04-26 13:33 /tmp/hadoop-yarn/staging/history/done drwxrwxrwt - mapred supergroup 0 2013-04-26 13:33 /tmp/hadoop-yarn/staging/history/done_intermediate kevin@devUbuntu05:~$ also please share exception stack trace... There is no exception now that I created /tmp on HDFS. But I still cannot see the logs via port 50030 on the master. In other words nothing seems to be listening on http:devubuntu05:50030. The log for map reduce looks like: 2013-04-26 13:35:26,107 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryClientService: Instantiated MRClientService at devUbuntu05/172.16.26.68:10020 2013-04-26 13:35:26,107 INFO org.apache.hadoop.yarn.service.AbstractService: Service:HistoryClientService is started. 2013-04-26 13:35:26,107 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer is started. 2013-04-26 13:35:55,290 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started 2013-04-26 13:35:55,295 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete 2013-04-26 13:38:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files 2013-04-26 13:41:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files 2013-04-26 13:44:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files 2013-04-26 13:47:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files 2013-04-26 13:50:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files Thanks, Omkar Joshi Hortonworks Inc On Fri, Apr 26, 2013 at 3:05 PM, <[email protected]> wrote: I was able to overcome the permission exception in the log by creating an HDFS tmp folder (hadoop fs -mkdir /tmp) and opening it up to the world (hadoop fs -chmod a+rwx /tmp). That got rid of the exception put I still am able to connect to port 50030 to see M/R status. More ideas? Even though the exception was missing from the logs of one server in the cluster, l looked on another server and found essentially the same permission problem: 2013-04-26 13:34:56,462 FATAL org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting JobHistoryServer org.apache.hadoop.yarn.YarnException: Error creating done directory: [hdfs://devubuntu05:9000/tmp/hadoop-yarn/staging/history/done] at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.init(HistoryFileManager .java:424) at org.apache.hadoop.mapreduce.v2.hs.JobHistory.init(JobHistory.java:87) at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:5 8) . . . . . On Fri, Apr 26, 2013 at 10:37 AM, Rishi Yadav wrote: do you see "retired jobs" on job tracker page. There is also "job tracker history" on the bottom of page. something like this http://nn.zettabyte.com:50030/jobtracker.jsp Thanks and Regards, Rishi Yadav On Fri, Apr 26, 2013 at 7:36 AM, < [email protected]> wrote: When I submit a simple "Hello World" M/R job like WordCount it takes less than 5 seconds. The texts show numerous methods for monitoring M/R jobs as they are happening but I have yet to see any that show statistics about a job after it has completed. Obviously simple jobs that take a short amount of time don't allow time to fire up any web mage or monitoring tool to see how it progresses through the JobTracker and TaskTracker as well as which node it is processed on. Any suggestions on how could see this kind of data *after* a job has completed?
