I dont see yyyy-mm-dd directory names but rather series of numbers
/user/hive/tmp/hive-testuser/248737551 <dir> 2010-04-11 16:36
rwxr-xr-x user supergroup
/user/hive/tmp/hive-testuser/334926967 <dir> 2010-04-11 16:36
rwxr-xr-x user supergroup
/user/hive/tmp/hive-testuser/376386234 <dir> 2010-04-11 16:36
rwxr-xr-x user supergroup
/user/hive/tmp/hive-testuser/773408034 <dir> 2010-04-11 16:36
rwxr-xr-x user supergroup
/user/hive/tmp/hive-testuser/855926514 <dir> 2010-04-11 16:36
rwxr-xr-x user supergroup
/user/hive/tmp/hive-testuser/plan.-1493385873 <r 3> 52634
2010-04-11 16:36 rw-r--r-- testuser supergroup
/user/hive/tmp/hive-testuser/plan.-1975696128 <r 3> 35389
2010-04-11 16:36 rw-r--r-- testuser supergroup
/user/hive/tmp/hive-testuser/plan.-508829937 <r 3> 21375
2010-04-11 16:36 rw-r--r-- testuser supergroup
/user/hive/tmp/hive-testuser/plan.1732465416 <r 3> 32861
2010-04-11 16:36 rw-r--r-- testuser supergroup
Thanks
Ashwin
On 5/11/10 1:34 PM, Namit Jain wrote:
The scratch dir has a format yyyy-mm-dd etc.
You can write a script to delete old data (say older than 2 days)
-----Original Message-----
From: Ashwin Agate [mailto:[email protected]]
Sent: Tuesday, May 11, 2010 1:31 PM
To: [email protected]
Subject: Re: Is hive.exec.scratchdir cleaned up ?
Hi Ning,
Thanks! Did verify that files do get deleted from the scratch directory
- I am probably seeing remnants from jobs that didnt finish normally
over the period of several weeks
Anyone knows of a smart technique of deleting those? (based on timestamp)
-A
On 5/11/10 1:01 PM, Ning Zhang wrote:
It should be cleaned up if the job finished normally or be killed by the
command printed out by CLI or through JT. If the CLI is killed by ^C or the JT
is killed etc the scratch dir won't be able to clean up.
On May 11, 2010, at 12:54 PM, Ashwin Agate wrote:
Hi
I see that hive.exec.scratchdir doesnt get cleaned up in hdfs.
Using hive checked out from svn built around feb 15.
Using Hive CLI to run a series of hive jobs and my test cluster doesnt
have much disk space to spare
Am I missing anything or has this been fixed in later versions of hive ?
Thanks,
A
