On Wed, Jul 21, 2010 at 3:01 PM, Allen Wittenauer <awittena...@linkedin.com>wrote:
> > On Jul 21, 2010, at 12:45 PM, Travis Crawford wrote: > > Any tips on how to avoid getting into this situation? > > What we started to do was two-fold: > > a) During every maintenance, we'd blow away the mapred temp dirs. The > TaskTracker does a very bad job of cleaning up after jobs and there is > usually a lot of cruft. If you have a 'flat' disk/fs structure such that MR > temp and HDFS is shared, this is a huge problem. > > I setup a cron job to delete files older than 5 days in mapred.local.dir. I've also found that sometimes userlogs aren't cleaned up correctly, so setting up a cron job to delete old files in userlogs is also a good idea. Good luck, Travis! Alex
