HBase's notion of what mapfiles are in a Region is gotten at region open
time. HBase and HDFS can diverge if the region has been opened
elsewhere -- possible during times of churn such as startup or after
regionserver crash (see hbase-1104). The open in other location could
have removed the file. Is this whats happening with you Dru? For the
below missing file, the regionid code is 2022550034. Grep the master
log file with '2022550034'. This should turn up the actual region name
('2022550034' is jenkins encoding of actual name). Then take the actual
region name and grep the master log. This'll emit the regions history.
Try studying it to see if you see it being assigned to two
regionservers. If no close in between, then you are probably suffering
hbase-1104 (There is also the region historian pages in the master UI if
you want to avoid logs; grep the encoded region name in the table page
in UI).
Usually a restart 'fixes' things so hbase has a 'true' view of whats on
the filesystem again but sounds like its not happening for you. Thats
strange. Its always the same file missing that is causing your MR to fail?
St.Ack
Dru Jensen wrote:
I have a MR process that is failing at the same point everytime I try
to run it. I have restarted hadoop and hbase but that didn't fix it
(unlike last time).
I checked dfs and the file that hbase is looking for does not exist.
Is there a tool I can run to fix this?
Thanks,
Dru
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server 10.10.10.52:60020 for region
talentmaps,http://reporternews.com/news/2008/feb/13/sampson-in-trouble-over-major-violations/,1226767718516,
row
'http://reporternews.com/news/2008/feb/13/sampson-in-trouble-over-major-violations/',
but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed
construction
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:88)
at
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2123)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1984)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1178)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1697)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://dev-hadoop0.brandaffinity.net:9000/hbase/talentmaps/2022550034/base/mapfiles/7007481172739780749/data
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:679)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
at
org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
at
org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
at
org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
at
org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 11 more
