The scratch dir has a format yyyy-mm-dd etc. You can write a script to delete old data (say older than 2 days)
-----Original Message----- From: Ashwin Agate [mailto:[email protected]] Sent: Tuesday, May 11, 2010 1:31 PM To: [email protected] Subject: Re: Is hive.exec.scratchdir cleaned up ? Hi Ning, Thanks! Did verify that files do get deleted from the scratch directory - I am probably seeing remnants from jobs that didnt finish normally over the period of several weeks Anyone knows of a smart technique of deleting those? (based on timestamp) -A On 5/11/10 1:01 PM, Ning Zhang wrote: > It should be cleaned up if the job finished normally or be killed by the > command printed out by CLI or through JT. If the CLI is killed by ^C or the > JT is killed etc the scratch dir won't be able to clean up. > > On May 11, 2010, at 12:54 PM, Ashwin Agate wrote: > > >> Hi >> >> I see that hive.exec.scratchdir doesnt get cleaned up in hdfs. >> >> Using hive checked out from svn built around feb 15. >> >> Using Hive CLI to run a series of hive jobs and my test cluster doesnt >> have much disk space to spare >> >> Am I missing anything or has this been fixed in later versions of hive ? >> >> Thanks, >> A >> > >
