Hi all,
I downloaded the latest release candidate (0.18.0 - with hadoop 0.18.0) to
evaluate whether it would suit my purposes and to get more familiar with the
software. I'm running on a cluster with 1 master and 4 region servers.
I was running some simple tests yesterday, inserting records into a table
and reading them out (largely based on the example in the API).
Today when I went to access the the table I was getting some exceptions. The
following was from my little test app that reads / writes to HBase:
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
to locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:789)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:432)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:442)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:446)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:407)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:101)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:86)
I also tried to do a describe of the table using the shell and this was the
result:
NativeException: java.lang.NullPointerException: null
from java/lang/String.java:433:in `<init>'
from org/apache/hadoop/hbase/util/Bytes.java:75:in `toString'
from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
`getMessage'
from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
`<init>'
from org/apache/hadoop/hbase/client/HConnectionManager.java:863:in
`getRegionServerWithRetries'
from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
from org/apache/hadoop/hbase/client/HConnectionManager.java:297:in
`listTables'
from org/apache/hadoop/hbase/client/HBaseAdmin.java:117:in `listTables'
from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:585:in `invoke'
from org/jruby/javasupport/JavaMethod.java:250:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
from org/jruby/javasupport/JavaClass.java:416:in `execute'
... 133 levels...
from
ruby.data1.home.fast.tools.hbase.hbase_minus_0_dot_18_dot_0.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in
`call'
from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
from org/jruby/runtime/CallSite.java:298:in `call'
from
ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:351:in
`__file__'
from
ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in
`__file__'
from
ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in
`load'
from org/jruby/Ruby.java:512:in `runScript'
from org/jruby/Ruby.java:432:in `runNormally'
from org/jruby/Ruby.java:312:in `runFromMain'
from org/jruby/Main.java:144:in `run'
from org/jruby/Main.java:89:in `run'
from org/jruby/Main.java:80:in `main'
from /home/fast/tools/hbase/current/bin/../bin/hirb.rb:246:in `describe'
I can see the data on HDFS through the browser, but for some reason I can't
get to it.
A short time later I noticed that my region processes had died (after I had
run the describe). I went back through the logs and noticed the following
repeated error that started yesterday:
2008-09-18 16:03:34,834 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
Rolling hlog. Number of entries: 51
2008-09-18 16:03:34,834 ERROR
org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
java.io.IOException: Could not get block locations. Aborting...
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
then finally:
2008-09-19 09:28:02,477 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 0)
java.io.IOException: java.io.IOException: java.lang.NullPointerException
at java.lang.String.<init>(String.java:433)
at org.apache.hadoop.hbase.util.Bytes.toString(Bytes.java:75)
at
org.apache.hadoop.hbase.client.RetriesExhaustedException.getMessage(RetriesExhaustedException.java:50)
at
org.apache.hadoop.hbase.client.RetriesExhaustedException.<init>(RetriesExhaustedException.java:40)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at
org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:237)
at
org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:216)
at
org.apache.hadoop.hbase.RegionHistorian.addRegionAssignment(RegionHistorian.java:140)
at
org.apache.hadoop.hbase.master.RegionManager.assignRegionsToMultipleServers(RegionManager.java:263)
at
org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:199)
at
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:367)
at
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:313)
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:232)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:564)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:494) at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:399)
at java.lang.Thread.run(Thread.java:595)
2008-09-19 09:28:02,478 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
master for 1993183 milliseconds - aborting server
A restart of hbase does not seem to have resolved the issue either. I get
the following error on the webpage when I try to access my table:
HTTP ERROR: 500
java.io.IOException: Could not obtain block:
blk_-1560243342111364142_23058
file=/hbase/-ROOT-/70236052/info/mapfiles/293988224693195769/data
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462)
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312)
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417)
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1369)
at java.io.DataInputStream.readInt(DataInputStream.java:353)
at
org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1895)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1925)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1830)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:516)
at
org.apache.hadoop.hbase.regionserver.HStore.rowAtOrBeforeFromMapFile(HStore.java:1504)
at
org.apache.hadoop.hbase.regionserver.HStore.getRowKeyAtOrBefore(HStore.java:1476)
at
org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1250)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1057)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
Could someone please advise as to what might be the problem and how to
resolve it.
Thanks,
Daniel
