Yeah, looks like the same thing. The job went well eventually (many mappers, lasted for 13h +, no errors at all).
Alex On Sun, Feb 12, 2012 at 4:35 PM, Mikael Sitruk <[email protected]>wrote: > I had same wired problem last week when i upgraded from cdh3u0 to cdh3u2. > I got lease exception even when the hbase.regionserver.lease.period was > set '300000'. The lease occurred after 1 mins or more. > Prior the upgrade everything was fine, i was executing with map reduce 8 > slots. > The lease totally disappeared when i use only 1 slot, i thought there were > some concurrency problem with MR integration, but this weekend some of the > machines where restarted and suddenly everything is fine!! I'm back to 8 > slot and the extract are running. (i run several time even without even one > lease exception) > In my case NO region server were stuck in transition. > It is really frustrating that i cannot point on what was the real problem. > Even log with debug did not point on problems (perhaps because it is also > missing some debug statement like when a scanner lease is added to the RS) > > Mikael.S > > > On Sun, Feb 12, 2012 at 9:55 PM, Alex Baranau <[email protected] > >wrote: > > > Hm, weird. Looks like now it goes well. What I did is once again > restarted > > several regionservers (noticed that some where holding regions which were > > stuck in "in transition" state). And started back the job. > > > > Btw, forgot to mention that caching on Scanner was set to only 64 and I > the > > estimate for records size is < 100KB. Anyhow, the weird thing was that > the > > limits were set to 10min and it started to fail even before 10 min has > > passed (sometimes after 3-4 minutes). > > > > Have to go and can't watch the job to the end, will write whether it > > succeed or not. If face same problems again will try brutal variant: set > > caching = 10 (or even 1), set batch = 10 (or even 1). > > > > Alex > > > > On Sun, Feb 12, 2012 at 1:49 PM, Alex Baranau <[email protected] > > >wrote: > > > > > Hi, > > > > > > 0.90.4-cdh3u2 > > > > > > Alex > > > > > > > > > On Sun, Feb 12, 2012 at 1:44 PM, <[email protected]> wrote: > > > > > >> Which version of hbase are you using ? > > >> > > >> Thanks > > >> > > >> > > >> > > >> On Feb 12, 2012, at 10:41 AM, Alex Baranau <[email protected]> > > >> wrote: > > >> > > >> > Hello, > > >> > > > >> > I'm getting scanner lease exceptions during mapreduce job [1] after > > >> running > > >> > it for less than 7 minutes. Though I have set > > >> > hbase.regionserver.lease.period to 600000 (i.e. 10 min) in hbase > > >> > configuration on master and all regionservers and master (and > > restarted > > >> > all). Also set it in job's configuration. > > >> > > > >> > Is there anything else I could do to eliminate the exception? > > >> > > > >> > Thanks, > > >> > Alex > > >> > > > >> > [1] > > >> > > > >> > org.apache.hadoop.hbase.regionserver.LeaseException: > > >> > org.apache.hadoop.hbase.regionserver.LeaseException: lease > > >> > ʼ3854333525977315054ʼ does not exist > > >> > at > > >> > > > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230) > > >> > > > >> > at > > >> > > > >> > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1862) > > >> > > > >> > at > > sun.reflect.GeneratedMethodAccessor6.invoke(Unknown > > >> > Source) > > >> > > > >> > at > > >> > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > >> > Last > > >> > at > > >> > java.lang.reflect.Method.invoke(Method.java:597) > > >> > 4KB > > >> > al FAILED at > > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > > >> > Last > > >> > at > > >> > > > >> > > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) > > >> > 8KB > > >> > > > >> > All > > >> > at > > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > >> > Method) > > >> > > > >> > at > > >> > > > >> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > >> > > > >> > at > > >> > > > >> > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > >> > > > >> > at > > >> > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > >> > > > >> > at > > >> > org.ap > > >> > > > > > > > > > > > > -- > Mikael.S >
