We use hbase 0.20.1 There are 3 region servers. 13 regions on each server. I don't see any exception in master log.
I was able to run 10 successful get commands before hitting the following: I am attaching (partial) master log and region server log from 10.10.31.137 hbase(main):014:0> get 'ruletable', 'com.about.acne' COLUMN CELL lpm_1.0:category timestamp=1268347483823, value= http://acne.about.com\t9002:0.86580086\thost\n 1 row(s) in 0.0120 seconds hbase(main):015:0> get 'ruletable', 'com.about.acne' NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.10.31.137:60020 for region ruletable,,1268083966723, row 'com.about.acne', but failed after 5 attempts. Exceptions: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: ruletable,,1268083966723 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) On Fri, Mar 12, 2010 at 9:08 PM, Jonathan Gray <jl...@streamy.com> wrote: > Seems like something weird is going on with your regionservers and > balancing. > > Can you post big snippets from the regionserver and master logs? Have you > checked to see what's going on? Is there repetitive balancing going on > that > never seems to reach steady-state? > > How many regions and how many nodes on which version of HBase? > > > -----Original Message----- > > From: Ted Yu [mailto:yuzhih...@gmail.com] > > Sent: Friday, March 12, 2010 8:24 PM > > To: hbase-user@hadoop.apache.org > > Subject: slow response in hbase shell > > > > Hi, > > > > > We sometimes saw over 5 second delay running get in hbase 0.20.1 > > shell: > > > > > > hbase(main):002:0> get 'ruletable', 'ca.tsn.www' > > > 0 row(s) in 10.1330 seconds > > > > > > From our 3 region servers there are a lot of such messages: > > > > > > > > > > 2010-03-12 00:00:00,996 INFO [regionserver/10.10.31.135:60020] > > > regionserver.HRegionServer(493): MSG_REGION_CLOSE: > > > > > crawltable,com.pandora.www:http\x2Finclude\x2FlyricsAdEmbed.html\x3Fgen > > re\x3Delectronica\x26artist\x3DR273847\x26webname\x3D\x26sz\x3D2000x8\x > > 26ord\x3D125823029226371645\x26tile\x3D3\x26_id\x3Dbottom_leaderboard_c > > ontainer,1266944566406: > > > Overloaded > > > 2010-03-12 00:00:00,997 INFO [regionserver/10.10.31.135:60020] > > > regionserver.HRegionServer(493): MSG_REGION_CLOSE: > > > domaincrawltable,,1268098564908: Overloaded > > > > > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > > > 40428 343638 11523230 > > > > > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > > > 40430 343655 11307703 > > > > > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > > > 40466 343961 11340379 > > > > > > Was the slow response due to the load balancing ? > > > > > > > The strange thing was that after several quick responses I would see: > > > > hbase(main):004:0> get 'ruletable', 'com.about.acne' > > COLUMN > > CELL > > lpm_1.0:category timestamp=1268347483823, value= > > http://acne.about.com\t9002:0.86580 086\thost\n > > 1 row(s) in 0.0040 seconds > > hbase(main):005:0> get 'ruletable', 'com.about.acne' > > COLUMN > > CELL > > lpm_1.0:category timestamp=1268347483823, value= > > http://acne.about.com\t9002:0.86580 086\thost\n > > 1 row(s) in 0.0040 seconds > > hbase(main):006:0> get 'ruletable', 'com.about.acne' > > NativeException: > > org.apache.hadoop.hbase.client.RetriesExhaustedException: > > Trying to contact region server 10.10.31.136:60020 for region > > ruletable,,1268083966723, row 'com.about.acne', but failed after 5 > > attempts. > > Exceptions: > > org.apache.hadoop.hbase.NotServingRegionException: > > org.apache.hadoop.hbase.NotServingRegionException: > > ruletable,,1268083966723 > > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionSer > > ver.java:2307) > > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja > > va:1784) > > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso > > rImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) > > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91 > > 5) > > > > But 006:60010/master.jsp refreshes quickly and shows all three > > regionservers. > > Don't know why hbase shell encountered NSRE. > > > > > > > > Thanks > >
