hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: After
restarting Datanode in HDFS
--------------------------------------------------------------------------------------------------------
Key: HBASE-1250
URL: https://issues.apache.org/jira/browse/HBASE-1250
Project: Hadoop HBase
Issue Type: Bug
Environment: Cluster is runing on one Laptop.
HDFS (1 Namenode + 1 Datanode)
HBase (1 master + 1 HRS)
Reporter: Evgeny Ryabitskiy
I am stoping Datanode.
And get a lot of exception. Then I am starting Datanode back.
Master is working again but Region Server returns exceptions:
INFO ipc.HBaseServer: IPC Server handler 9 on 60020, call
openScanner([...@e3404f, [...@1930611, [...@1735f84, 9223372036854775807, null)
from 127.0.0.1:55743: error: java.io.IOException: HStoreScanner failed
construction
java.io.IOException: HStoreScanner failed construction
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:73)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:87)
at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1769)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1971)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1161)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1720)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
Caused by: java.io.IOException: Could not obtain block:
blk_-5468922917120193795_1013
file=/hbase/-ROOT-/70236052/info/2088312954622080873
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
at java.io.DataInputStream.read(DataInputStream.java:132)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:910)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:863)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1151)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.getNext(StoreFileScanner.java:301)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.openScanner(StoreFileScanner.java:102)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)
... 10 more
this one is returned in a loop and there is second one:
WARN hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block:
blk_-5468922917120193795_1013
file=/hbase/-ROOT-/70236052/info/2088312954622080873
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
at java.io.DataInputStream.read(DataInputStream.java:132)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:910)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:863)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1151)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.getNext(StoreFileScanner.java:301)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.openScanner(StoreFileScanner.java:102)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:87)
at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1769)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1971)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1161)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1720)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
