On Wed, Jul 29, 2009 at 10:06 PM, Marc Limotte <[email protected]> wrote:
> A quick question about the user logs (hadoop 0.19.1). > > I've looked all over HDFS. I can't locate the actual userlogs. Where are > they stored? I can get to them through the job tracker web interface, but > I'd like to be able to script some jobs to check the logs for errors. I > suppose I could use curl against the web interface, but that seems like a > hack. Alternatively, is there command line access to the userlogs through > the hadoop command? I couldn't find anything like that. > Thanks for any tips... > > All hadoop logs are stored on the local filesystem on each node (the location is configurable). What you could do though is run a cron job which periodically pushes log files to HDFS at the end of the day. Here's another idea: http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-for-hadoop-log-collection/ -- Harish Mallipeddi http://blog.poundbang.in
