Regions failed compaction left in open and writable state but are not scannable
-------------------------------------------------------------------------------
Key: HBASE-1115
URL: https://issues.apache.org/jira/browse/HBASE-1115
Project: Hadoop HBase
Issue Type: Bug
Environment: apurtell cluster, HBase TRUNK on hadoop 0.18 branch
Reporter: Andrew Purtell
Priority: Critical
After an instance of HBASE-1114, where compaction fails, the affected region is
left in an open and writable state, but scanners fail construction. Later a
manual reassignment via close_region brings the region all the way back up.
Line numbers are for 0.19-on-hadoop-0.18 branch, sorry about that. I'm
upgrading to Hadoop 0.19 (finally) this afternoon.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 10.30.94.50:60020 for region
content,c84bbfc94b2143e41ba119d159be2958,1231518442461, row
'c84bbfc94b2143e41ba119d159be2958', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2119)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1878)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1162)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1673)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1707725801/url/mapfiles/7039742044868774100/data
at
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
at
org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
at
org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
at
org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
at
org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
at
org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
