Hi there:
On an internal testing cluster with 3 nodes, when I run "flush '.META' " on
the hbase shell, it gets the following "Can not obtain block" error. I
checked around, and many posts say that it might be due to the crash of some
datanodes. But in my case, i checked the UI, all nodes appears to be fine.
Any other possibilities?
Error details pasted below. Any help is appreciated!
Best,
Arber
hbase(main):001:0> flush '.META'
10/07/12 17:29:30 WARN client.HConnectionManager$TableServers: Testing for
table existence threw exception
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server null for region , row '', but failed after 5 attempts.
Exceptions:
java.io.IOException: java.io.IOException: Could not obtain block:
blk_-80326634570231114_202750
file=/hbase/-ROOT-/70236052/info/mapfiles/3687060941742211902/data
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1592)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1909)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.next(SequenceFile.java:1939)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.next(SequenceFile.java:1844)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.next(SequenceFile.java:1890)
at org.apache.hadoop.hbase.io.MapFile$Reader.next(MapFile.java:544)
at
org.apache.hadoop.hbase.regionserver.HStore.rowAtOrBeforeFromMapFile(HStore.java:1723)
at
org.apache.hadoop.hbase.regionserver.HStore.getRowKeyAtOrBefore(HStore.java:1695)
at
org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1089)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1555)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
