Andy and St.Ack,
Thanks for you help. The directory didn't exist and the MR process
failed in the same spot after restarting the cluster. I decided to
shutdown/restart and try again. This time it fixed itself so I guess
I was lucky.
Dru
On Jan 8, 2009, at 10:11 PM, stack wrote:
Andrew Purtell wrote:
Hi Dru,
You might try
% hadoop fs -rmr hdfs://dev-hadoop0.brandaffinity.net:9000/hbase/
talentmaps/2022550034/base/mapfiles/7007481172739780749/
Dru, per Andrew's note above, does the parent directory exist? If
so, and you list it, what do you see? No data but is there an index?
St.Ack
to at least partially recover the region. Don't try this until
someone else weighs in on this approach.
Since you've already suffered data loss, possibly as a result of
HBASE-1104, I do not think there is any harm to try it.
- Andy
From: Dru Jensen <[email protected]>
Subject: corrupt hbase meta
To: [email protected]
Date: Thursday, January 8, 2009, 4:48 PM
I have a MR process that is failing at the same point
everytime I try to run it. I have restarted hadoop and
hbase but that didn't fix it (unlike last time).
I checked dfs and the file that hbase is looking for does
not exist. Is there a tool I can run to fix this?
Thanks,
Dru
org.apache.hadoop.hbase.client.RetriesExhaustedException:
Trying to contact region server 10.10.10.52:60020 for region
talentmaps,http://reporternews.com/news/2008/feb/13/sampson-in-trouble-over-major-violations/,1226767718516
,
row
'http://reporternews.com/news/2008/feb/13/sampson-in-trouble-over-major-violations/'
,
but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner
failed construction
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.
(StoreFileScanner.java:70)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.
(HStoreScanner.java:88)
at
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:
2123)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.
(HRegion.java:1984)
at
org
.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:
1178)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:
1697)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun
.reflect
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun
.reflect
.DelegatingMethodAccessorImpl
.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer
$Handler.run(HBaseServer.java:894)
Caused by: java.io.FileNotFoundException: File does not
exist:
hdfs://dev-hadoop0.brandaffinity.net:9000/hbase/talentmaps/
2022550034/base/mapfiles/7007481172739780749/data
at
org
.apache
.hadoop
.hdfs
.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at
org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:679)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:
1431)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:
1426)
at
org.apache.hadoop.hbase.io.MapFile
$Reader.createDataFileReader(MapFile.java:310)
at
org.apache.hadoop.hbase.io.HBaseMapFile
$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
at
org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
at
org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.
(HBaseMapFile.java:79)
at
org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.
(BloomFilterMapFile.java:65)
at
org
.apache
.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
at
org
.apache
.hadoop
.hbase
.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.
(StoreFileScanner.java:67)
... 11 more
