Late answer, did you figure it out? This exception happens when you don't use your scanner lease for more than the lease time (default one minute). AFAIK that didn't change, so maybe something else got slow? Or maybe some special configurations you had didn't make it during the upgrade?
J-D On Mon, Feb 6, 2012 at 12:33 AM, Mikael Sitruk <[email protected]> wrote: > Hi all > > Recently I have upgraded my cluster from Hbase 0.90.1 to 0.90.4 (using > cloudera from cdh3u0 to cdh3u2) > Everything was ok till I ran pig extract on the new cluster, from the old > cluster everything worked well. > Now each time i run the extract in conjunction to other work performed on > the cluster I get the following exception during job execution, and the job > is finally killed. > It appears only on heavy extract (For small extract < minute execution > everything is ok). > > Any suggestion? > Below a stacktrace from the task. > > 2012-02-06 01:18:26,561 INFO > org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat: > setScan with ranges: 374419525347267552315334599026232694541505353300281 - > 374419525347267552315334599026232694541509698599225 ( 4345298944) > 2012-02-06 01:19:32,836 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > 2012-02-06 01:19:32,845 WARN org.apache.hadoop.mapred.Child: Error running > child org.apache.hadoop.hbase.regionserver.LeaseException: > org.apache.hadoop.hbase.regionserver.LeaseException: lease > '-7220618182832784549' > does not exist > > at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java: 230) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java: > 1862) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: > 25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java: 570) > at > org.apache.hadoop.hbase.ipc.HBaseServer $Handler.run(HBaseServer.java:1039) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java: > 39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java: > 27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java: > 96) > > at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java: > 83) > > at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java: > 38) > at > org.apache.hadoop.hbase.client.HConnectionManager > $HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java: > 1019) > at > org.apache.hadoop.hbase.client.HTable $ClientScanner.next(HTable.java:1151) > > at > org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java: > 133) > > at > org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java: > 142) > at > org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat > $HBaseTableRecordReader.nextKeyValue(HBaseTableInputFormat.java:162) > > at > org.apache.pig.backend.hadoop.hbase.HBaseStorage.getNext(HBaseStorage.java: > 321) > > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java: > 187) > at > org.apache.hadoop.mapred.MapTask > $NewTrackingRecordReader.nextKeyValue(MapTask.java:456) > > at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java: 67) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java: > 1127) > at org.apache.hadoop.mapred.Child.main(Child.java:264) 2012-02-06 > 01:19:32,978 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the > task > > Thanks > Mikael.S
