Daniel, Many of the errors I see seem related to the fact that something went wrong under the hood... But got an NPE will trying to process it (seems the region name was null hence no region found).
It seems to indicate that your data in HDFS went away. Have you changed Hadoop default config that puts the data into /tmp ? J-D On Thu, Sep 18, 2008 at 7:49 PM, Daniel Ploeg <[EMAIL PROTECTED]> wrote: > Hi all, > > I downloaded the latest release candidate (0.18.0 - with hadoop 0.18.0) to > evaluate whether it would suit my purposes and to get more familiar with > the > software. I'm running on a cluster with 1 master and 4 region servers. > > I was running some simple tests yesterday, inserting records into a table > and reading them out (largely based on the example in the API). > > Today when I went to access the the table I was getting some exceptions. > The > following was from my little test app that reads / writes to HBase: > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying > to locate root region > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:789) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:432) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:442) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:446) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:407) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:101) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:86) > > I also tried to do a describe of the table using the shell and this was the > result: > > NativeException: java.lang.NullPointerException: null > from java/lang/String.java:433:in `<init>' > from org/apache/hadoop/hbase/util/Bytes.java:75:in `toString' > from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in > `getMessage' > from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in > `<init>' > from org/apache/hadoop/hbase/client/HConnectionManager.java:863:in > `getRegionServerWithRetries' > from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan' > from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan' > from org/apache/hadoop/hbase/client/HConnectionManager.java:297:in > `listTables' > from org/apache/hadoop/hbase/client/HBaseAdmin.java:117:in `listTables' > from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0' > from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke' > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke' > from java/lang/reflect/Method.java:585:in `invoke' > from org/jruby/javasupport/JavaMethod.java:250:in > `invokeWithExceptionHandling' > from org/jruby/javasupport/JavaMethod.java:219:in `invoke' > from org/jruby/javasupport/JavaClass.java:416:in `execute' > ... 133 levels... > from > > ruby.data1.home.fast.tools.hbase.hbase_minus_0_dot_18_dot_0.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in > `call' > from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call' > from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call' > from org/jruby/runtime/CallSite.java:123:in `cacheAndCall' > from org/jruby/runtime/CallSite.java:298:in `call' > from > > ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:351:in > `__file__' > from > > ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in > `__file__' > from > > ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in > `load' > from org/jruby/Ruby.java:512:in `runScript' > from org/jruby/Ruby.java:432:in `runNormally' > from org/jruby/Ruby.java:312:in `runFromMain' > from org/jruby/Main.java:144:in `run' > from org/jruby/Main.java:89:in `run' > from org/jruby/Main.java:80:in `main' > from /home/fast/tools/hbase/current/bin/../bin/hirb.rb:246:in `describe' > > I can see the data on HDFS through the browser, but for some reason I can't > get to it. > > A short time later I noticed that my region processes had died (after I had > run the describe). I went back through the logs and noticed the following > repeated error that started yesterday: > > 2008-09-18 16:03:34,834 INFO > org.apache.hadoop.hbase.regionserver.LogRoller: > Rolling hlog. Number of entries: 51 > 2008-09-18 16:03:34,834 ERROR > org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed > java.io.IOException: Could not get block locations. Aborting... > at > > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143) > at > > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735) > at > > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889) > > then finally: > > 2008-09-19 09:28:02,477 WARN > org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message > (Retry: 0) > java.io.IOException: java.io.IOException: java.lang.NullPointerException > at java.lang.String.<init>(String.java:433) > at org.apache.hadoop.hbase.util.Bytes.toString(Bytes.java:75) > at > > org.apache.hadoop.hbase.client.RetriesExhaustedException.getMessage(RetriesExhaustedException.java:50) > at > > org.apache.hadoop.hbase.client.RetriesExhaustedException.<init>(RetriesExhaustedException.java:40) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863) > at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964) > at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950) > at > org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:237) > at > org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:216) > at > > org.apache.hadoop.hbase.RegionHistorian.addRegionAssignment(RegionHistorian.java:140) > at > > org.apache.hadoop.hbase.master.RegionManager.assignRegionsToMultipleServers(RegionManager.java:263) > at > > org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:199) > at > > org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:367) > at > > org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:313) > at > > org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:232) > at > org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:564) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at > org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:494) at > > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82) > at > > org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:399) > at java.lang.Thread.run(Thread.java:595) > 2008-09-19 09:28:02,478 FATAL > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to > master for 1993183 milliseconds - aborting server > > A restart of hbase does not seem to have resolved the issue either. I get > the following error on the webpage when I try to access my table: > > HTTP ERROR: 500 > > java.io.IOException: Could not obtain block: > blk_-1560243342111364142_23058 > file=/hbase/-ROOT-/70236052/info/mapfiles/293988224693195769/data > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462) > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312) > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417) > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1369) > at java.io.DataInputStream.readInt(DataInputStream.java:353) > at > org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1895) > at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1925) > at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1830) > at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876) > at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:516) > at > org.apache.hadoop.hbase.regionserver.HStore.rowAtOrBeforeFromMapFile(HStore.java:1504) > at > org.apache.hadoop.hbase.regionserver.HStore.getRowKeyAtOrBefore(HStore.java:1476) > at > org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1250) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1057) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at > org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) > > > Could someone please advise as to what might be the problem and how to > resolve it. > > Thanks, > Daniel >
