NPE during compaction/split breaks table
----------------------------------------
Key: HBASE-1046
URL: https://issues.apache.org/jira/browse/HBASE-1046
Project: Hadoop HBase
Issue Type: Bug
Components: regionserver
Reporter: Andrew Purtell
Fix For: 0.19.0
After reassignment of region split, scanners (mapred tasks) fail:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 10.30.94.49:60020 for region
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row
'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
at
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
at
org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
at
org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
at
org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
at
org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
at
org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
According to the master the regions around this row are:
content,c4580f7292ceb2f6c8d732d935302633,1228328680554
10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633
c4e518dc579ca98bf97ea13d1f7c80fc
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020
2031068422 c4e518dc579ca98bf97ea13d1f7c80fc
c528c8864e67a3addcb50587586dfe38
On 10.30.94.49, just before the HStoreScanner exceptions begin:
2008-12-04 16:18:30,538 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,539 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,539 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175
available
2008-12-04 16:18:30,633 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422
available
2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region
content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:36,673 ERROR
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for
region content,c528c8864e67a3addcb50587586dfe38,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:36,691 ERROR
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for
region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
The previous region server was asked by the master to close it:
2008-12-04 07:58:03,525 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_CLOSE: content,c4580f7292ceb2f6c8d732d935302633,1228328680554:
Overloaded
2008-12-04 07:58:03,525 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Worker: MSG_REGION_CLOSE:
content,c4580f7292ceb2f6c8d732d935302633,1228328680554: Overloaded
2008-12-04 07:58:03,526 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed content,c4580f7292ceb2f6c8d732d935302633,1228328680554
There are no exceptions or other indications of problems with the region or the
region server in the relevant log file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
