Please update your hbase from 0.20.1. Its not much fun for those
helping debug issues to discover, after having expended some effort
debugging, that the issue has already been fixed.
UnknownScannerException usually means the client has taken too long to
report back to the regionserver between next invocations or the
regionserver was stuck GC'ing longer than the scanner lease.
St.Ack
On Thu, Apr 8, 2010 at 11:03 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> Here are snippets from master log w.r.t. region
> domaincrawltable,,1270600690648:
> 2010-04-07 00:00:38,504 DEBUG [RegionManager.metaScanner]
> master.BaseScanner(385): GET on domaincrawltable,,1270600690648 got
> different startcode than SCAN: sc=1270602502182, serverAddress=1270597824201
> 2010-04-07 00:00:38,541 INFO [RegionManager.metaScanner]
> master.BaseScanner(224): RegionManager.metaScanner scan of 7 row(s) of meta
> region {server: 10.10.30.82:60020, regionname: .META.,,1, startKey: <>}
> complete
>
> 2010-04-07 18:19:37,384 DEBUG [HMaster] master.ProcessRegionOpen(98): Adding
> to onlineMetaRegions: {server: 10.10.30.82:60020, regionname: .META.,,1,
> startKey: <>}
> 2010-04-07 18:19:39,417 INFO [IPC Server handler 11 on 60000]
> master.ServerManager(440): Processing MSG_REPORT_PROCESS_OPEN:
> domaincrawltable,,1270600690648 from snvgold.pr.com,60020,1270689385704; 1
> of 2
> 2010-04-07 18:19:39,417 INFO [IPC Server handler 11 on 60000]
> master.ServerManager(440): Processing MSG_REPORT_OPEN:
> domaincrawltable,,1270600690648 from snvgold.pr.com,60020,1270689385704; 2
> of 2
> 2010-04-07 18:19:39,419 DEBUG [HMaster] master.HMaster(486): Processing
> todo: PendingOpenOperation from snvgold.pr.com,60020,1270689385704
> 2010-04-07 18:19:39,419 INFO [HMaster] master.ProcessRegionOpen(70):
> domaincrawltable,,1270600690648 open on 10.10.30.82:60020
> 2010-04-07 18:19:39,423 INFO [HMaster] master.ProcessRegionOpen(80):
> Updated row domaincrawltable,,1270600690648 in region .META.,,1 with
> startcode=1270689385704, server=10.10.30.82:60020
>
> We use hbase 0.20.1 on server and client.
> The most peculiar log from one of regionservers is:
>
> 2010-04-08 10:26:38,391 ERROR [IPC Server handler 61 on 60020]
> regionserver.HRegionServer(844):
> org.apache.hadoop.hbase.UnknownScannerException: Name: -1
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1925)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> On Thu, Apr 8, 2010 at 10:40 AM, Jean-Daniel Cryans
> <jdcry...@apache.org>wrote:
>
>> No it's there: domaincrawltable,,1270600690648
>>
>> J-D
>>
>> On Thu, Apr 8, 2010 at 10:38 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > What if there is no region information in NSRE ?
>> >
>> > 2010-04-08 10:26:38,385 ERROR [IPC Server handler 60 on 60020]
>> > regionserver.HRegionServer(846): Failed openScanner
>> > org.apache.hadoop.hbase.NotServingRegionException:
>> > domaincrawltable,,1270600690648
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1893)
>> > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>> >
>> >
>> > On Thu, Apr 8, 2010 at 9:39 AM, Jean-Daniel Cryans <jdcry...@apache.org
>> >wrote:
>> >
>> >> On Wed, Apr 7, 2010 at 11:38 PM, Al Lias <al.l...@gmx.de> wrote:
>> >> > Occationally my HTable clients get a response that no server is
>> serving
>> >> > a particular region...
>> >> > Normally, the region is back a few seconds later (perhaps a split?).
>> >>
>> >> Or the region moved.
>> >>
>> >> >
>> >> > Anyway, the client (Using HTablePool) seems to need a restart to
>> forget
>> >> > this.
>> >>
>> >> Seems wrong, would love a stack trace.
>> >>
>> >> >
>> >> > Is there a config value to manipulate the caching time of regionserver
>> >> > assignments in the client?
>> >>
>> >> Nope, when the client sees a NSRE, it queries .META. to find the new
>> >> location.
>> >>
>> >> >
>> >> > I set a small value for hbase.client.pause to get failures fast. I am
>> >> > using 0.20.3 .
>> >>
>> >> Splits are still kinda slow, takes at least 2 seconds to happen, but
>> >> finding the new location of a region is a core feature in HBase and
>> >> it's rather well tested, Can you pin down your exact problem? Next
>> >> time a NSRE happens, see which region it was looking for and grep the
>> >> master log for it, you should see the history and how much time it
>> >> took to move.
>> >>
>> >> >
>> >> > Thx,
>> >> >
>> >> > Al
>> >> >
>> >>
>> >
>>
>
