You could take a look at Chukwa, which essentially collects and drops your logs to HDFS: <http://wiki.apache.org/hadoop/Chukwa>
The last time I tried to play with Chukwa, it wasn't in a state to be played with yet. If that's still the case, then you can use Scribe to collect all of your logs in a single place, and then create a quick Python script to persist these logs to HDFS. Learn more about Scribe here: < http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-for-hadoop-log-collection/ > Alex On Tue, Nov 18, 2008 at 2:37 PM, Nathan Marz <[EMAIL PROTECTED]> wrote: > We find that after about 400 to 500 jobs run in succession on our Hadoop > cluster, the disk space on each machine is quickly used up by logs for all > the tasks. What do people do to manage these logs? Does Hadoop have anything > built in for managing them? Or do we have to delete/move the logs with a > home-cooked method? > > Thanks, > Nathan Marz > Rapleaf >
