Two of our proprietary servers were trying to create ruletable at the same
time.
Here is log from region server 1:
2010-03-12 13:56:29,687 DEBUG [pool-1-thread-1] hfile.LruBlockCache(551):
Cache Stats: Sizes: Total=10.074425MB (10563800), Free=1217.9005MB
(1277061288), Max=1227.975MB (1287625088), Counts: Blocks=0, Access=0,
Hit=0, Miss=0, Evictions=0, Evicted=0, Ratios: Hit Ratio=NaN%, Miss
Ratio=NaN%, Evicted/Run=NaN
2010-03-12 13:56:56,030 INFO [regionserver/10.10.31.17:60020]
regionserver.HRegionServer(493): MSG_REGION_OPEN: ruletable,,1268431015006
2010-03-12 13:56:56,031 INFO [regionserver/10.10.31.17:60020.worker]
regionserver.HRegionServer$Worker(1485): Worker: MSG_REGION_OPEN:
ruletable,,1268431015006
2010-03-12 13:56:56,032 DEBUG [regionserver/10.10.31.17:60020.worker]
regionserver.HRegion(264): Opening region ruletable,,1268431015006,
encoded=169631626
2010-03-12 13:56:56,065 INFO [regionserver/10.10.31.17:60020.worker]
regionserver.HRegion(342): region ruletable,,1268431015006/169631626
available; sequence id is 0
2010-03-12 13:57:29,687 DEBUG [pool-1-thread-1] hfile.LruBlockCache(551):
Cache Stats: Sizes: Total=10.074425MB (10563800), Free=1217.9005MB
(1277061288), Max=1227.975MB (1287625088), Counts: Blocks=0, Access=0,
Hit=0, Miss=0, Evictions=0, Evicted=0, Ratios: Hit Ratio=NaN%, Miss
Ratio=NaN%, Evicted/Run=NaN
2010-03-12 13:57:33,223 ERROR [IPC Server handler 1 on 60020]
regionserver.HRegionServer(844):
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
family lpm_1.0 does not exist in region ruletable,,1268431015006 in table
{NAME => 'ruletable', FAMILIES => []}
at
org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:2375)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1241)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Here is log from region server 2:
2010-03-12 14:42:29,742 ERROR [IPC Server handler 30 on 60020]
regionserver.HRegionServer(846): Failed openScanner
java.lang.IllegalArgumentException
at java.util.PriorityQueue.<init>(PriorityQueue.java:135)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.<init>(KeyValueHeap.java:55)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:1695)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1073)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1054)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1894)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-03-12 14:42:29,751 ERROR [IPC Server handler 31 on 60020]
regionserver.HRegionServer(844):
org.apache.hadoop.hbase.UnknownScannerException: Name: -1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1925)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-03-12 14:43:02,787 INFO [regionserver/10.10.31.18:60020]
regionserver.HRegionServer(493): MSG_REGION_CLOSE: ruletable,,1268431015006
2010-03-12 14:43:02,788 INFO [regionserver/10.10.31.18:60020.worker]
regionserver.HRegionServer$Worker(1485): Worker: MSG_REGION_CLOSE:
ruletable,,1268431015006
2010-03-12 14:43:02,789 DEBUG [regionserver/10.10.31.18:60020.worker]
regionserver.HRegion(453): Closing ruletable,,1268431015006: compactions &
flushes disabled
2010-03-12 14:43:02,789 DEBUG [regionserver/10.10.31.18:60020.worker]
regionserver.HRegion(470): Updates disabled for region, no outstanding
scanners on ruletable,,1268431015006
2010-03-12 14:43:02,789 DEBUG [regionserver/10.10.31.18:60020.worker]
regionserver.HRegion(477): No more row locks outstanding on region
ruletable,,1268431015006
2010-03-12 14:43:02,789 INFO [regionserver/10.10.31.18:60020.worker]
regionserver.HRegion(489): Closed ruletable,,1268431015006
2010-03-12 14:43:11,849 INFO [regionserver/10.10.31.18:60020]
regionserver.HRegionServer(493): MSG_REGION_FLUSH: .META.,,1
2010-03-12 14:43:11,849 INFO [regionserver/10.10.31.18:60020]
regionserver.HRegionServer(493): MSG_REGION_MAJOR_COMPACT: .META.,,1
2010-03-12 14:43:11,850 INFO [regionserver/10.10.31.18:60020.worker]
regionserver.HRegionServer$Worker(1485): Worker: MSG_REGION_FLUSH: .META.,,1
2010-03-12 14:43:11,850 DEBUG [regionserver/10.10.31.18:60020.worker]
regionserver.HRegion(894): Started memstore flush for region .META.,,1.
Current region memstore size 168.0
On Fri, Mar 12, 2010 at 3:55 PM, Jean-Daniel Cryans <jdcry...@apache.org>wrote:
> Ted,
>
> You aren't the first one to report that issue (I saw 2 other ppl
> reporting it since 2 weeks ago), looks like a real bug. Can you grep
> around your hbase logs for ruletable,,1268431015006 and see if there's
> any exception related to that region? Can you identify exactly when it
> happened and what was happening?
>
> Thx
>
> J-D
>
> On Fri, Mar 12, 2010 at 2:33 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> > Hi,
> > When I tried to insert into ruletable, I saw:
> >
> > hbase(main):003:0> put 'ruletable', 'com.yahoo.www', 'lpm_1.0:category',
> > '1123:1' NativeException:
> > org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException:
> > org.apache.hadoop.hba
> > se.regionserver.NoSuchColumnFamilyException: Column family lpm_1.0 does
> not
> > exist in region ruletable,,1 268431015006 in table {NAME =>
> > 'ruletable', FAMILIES => []}
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:2375)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1241)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> >
> > However:
> > hbase(main):002:0> describe 'ruletable'
> > DESCRIPTION
> > {NAME => 'ruletable', FAMILIES => [{NAME => 'exactmatch_1.0', VERSIONS
> =>
> > '3', COMPRESSION => 'LZO', TTL => '1209600', TTU => '1123300', BLOCKSIZE
> =>
> > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
> > {NAME => 'lpm_1.0', COMPRESSION => 'LZO', VERSIONS => '3', TTL =>
> > '15552000', TTU => '14688000', BLOCKSIZE => '65536', IN_MEMORY =>
> 'false',
> > BLOCKCACHE => 'true'}]}
> >
> > Can someone explain the above scenario ?
> >
> > Thanks
> >
>
