Re: dynamic partition query dies with LeaseExpiredException

2010-10-01 Thread Ning Zhang
Hi Dave, There may be 2 issues here: 1) according to the HDFS JIRA it seems to be a bug introduced in Hadoop 0.18 and was fixed by another hadoop JIRA in 0.20.2 suggested by Hairong, you may want to try 0.20.2 if that's possible. 2) the fact that the lease expired seems to be caused by the

Re: dynamic partition query dies with LeaseExpiredException

2010-10-01 Thread Dave Brondsema
Thanks Ning, Our TRANSFORM script is pretty quick, the whole query only took 30 seconds when we added and project='abc' to limit the amount of input data. So it seems like the slowdown is somewhere in handling the data before (or after?) it gets to the transform script. I can try adding some