Hello!
It seems I have a serious problem with HBase. We've been using if for
some time now, had some minor problems with regions unexpectedly closing
once every couple of days, but this would usually sort itself out after
a couple of stops and starts. The hbase runs atop of a cluster of 3
machines. One of them is running HMaster and regionserver the other two
are regionservers .The replication is 3.
So while after restarting hbase this morning, I realize I cant get any
record out of it - a 'get' for a random record in hbase shell returns this:
hbase(main):001:0> get 'page-repository',
'http://sws.geonames.org/3623940/'
NativeException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying
to contact region server 140.203.154.174:60020 for region
page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058,
row 'http://sws.geonames.org/3623940/', but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1436)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1077)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1053)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1436)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1077)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1053)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1436)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1077)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1053)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
from
org/apache/hadoop/hbase/client/HConnectionManager.java:885:in
`getRegionServerWithRetries'
from
org/apache/hadoop/hbase/client/HTable.java:546:in `getRow'
from
org/apache/hadoop/hbase/client/HTable.java:507:in `getRow'
from sun/reflect/NativeMethodAccessorImpl.java:-2:in
`invoke0'
from sun/reflect/NativeMethodAccessorImpl.java:39:in
`invoke'
from
sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:250:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:219:in
`invoke'
from org/jruby/javasupport/JavaClass.java:416:in
`execute'
from
org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
`call'
from
org/jruby/internal/runtime/methods/DynamicMethod.java:78:in `call'
from org/jruby/runtime/CallSite.java:155:in
`cacheAndCall'
from org/jruby/runtime/CallSite.java:332:in `call'
from org/jruby/evaluator/ASTInterpreter.java:649:in
`callNode'
from org/jruby/evaluator/ASTInterpreter.java:324:in
`evalInternal'
... 129 levels...
from
ruby.sindice.hbase_minus_0_dot_2_dot_0.bin.hirbInvokermethod__22$RUBY$startOpt:-1:in
`call'
from
org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
from
org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
from org/jruby/runtime/CallSite.java:123:in
`cacheAndCall'
from org/jruby/runtime/CallSite.java:298:in `call'
from
ruby/sindice/hbase_minus_0_dot_2_dot_0/bin//home/sindice/hbase/bin/../bin/hirb.rb:338:in
`__file__'
from
ruby/sindice/hbase_minus_0_dot_2_dot_0/bin//home/sindice/hbase/bin/../bin/hirb.rb:-1:in
`__file__'
from
ruby/sindice/hbase_minus_0_dot_2_dot_0/bin//home/sindice/hbase/bin/../bin/hirb.rb:-1:in
`load'
from org/jruby/Ruby.java:512:in `runScript'
from org/jruby/Ruby.java:432:in `runNormally'
from org/jruby/Ruby.java:312:in `runFromMain'
from org/jruby/Main.java:144:in `run'
from org/jruby/Main.java:89:in `run'
from org/jruby/Main.java:80:in `main'
from /home/sindice/hbase/bin/../bin/hirb.rb:255:in `get'
from (hbase):2:in `binding'
At the Same Time, one regionserver (which one depends on the requested
field i guess) showers his logs with this sort of messages
2008-08-05 13:33:51,768 INFO
org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020,
call getRow([EMAIL PROTECTED], [EMAIL PROTECTED], 9223372036854775807)
from
140.203.154.174:35053: error:
org.apache.hadoop.hbase.NotServingRegionException:
page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
org.apache.hadoop.hbase.NotServingRegionException:
page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1436)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1077)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1053)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
What can I do to get rid of this problem and have HBase get back to work?
Thank you very much for your time,
Regards,
Nickolai
