g00dn3ss wrote:
....Until HBase started failing, I
didn't even consider that HBase would have problems with me removing corrupt
files.
HBase reads all the files in an HStore on open of a region. It keeps
the list of files in memory and doesn't expect it to change over a
HStore's lifecycle.
If for any reason -- hdfs hiccup -- the list that hbase has in memory
strays from whats on the filesystem because the filesystem has lost a
file or lost a block in a file, then hbase will have trouble with the
damaged Store. You'll see the issue on an access, either a fetch or
when hbase goes to compact the damaged Store. Currently, to get hbase
to reread the filesystem, the region needs to be redeployed. This means
restart of the hosting RegionServer (on the particular host run
'./bin/hbase regionserver stop' and then start). A whole regionserver
restart can be disruptive especially if the cluster is small and the
regionserver is carrying lots of regions. A new tool was added to the
shell in TRUNK that allows you close an individual region (In the shell,
type 'tools'). On region close it'll be redeployed elsewhere by the
master. On reopen, hbase will reread the filesystem content.
Alternately, is
there a way that I can manually initialize the corrupt table regions to
empty?
In the shell there is a truncate for the whole table.
Otherwise, if you would operate at the region-level only, first try the
above trick redeploying the region. If that doesn't work, read the log
for the problematic file. Is it present? If so, if you just want to
get going again, remove the bad file and then redeploy the region
again. Use the ./bin/hadoop fsck /HBASE_ROOT to help you figure problem
files. Cycle till you've cleared all corruption.
On Thu, Dec 25, 2008 at 4:26 AM, Andrew Purtell <[email protected]> wrote:
In general, I think it may be useful for HBase to provide
a recovery option where a corrupt table region can be
reinitialized as empty. At least the whole table will not
be lost. I have wanted something like this on occasion.
This could be a new shell tool.
Maybe one at the store level too? So, clean_store and clean_region?
One thing you can do is schedule daily maintenance time
where you shut down your cluster and do a Hadoop distcp
from the HBase/primary cluster to a secondary DFS cluster
serving as backup media. This is akin to making a tape
backup and has the same drawback of losing all edits
subsequent to the last backup upon recovery, but on the
other hand you do not lose everything. The distcp copies
the data in reasonable parallel fashion so the backup
can complete quickly even if the tables are large.
It would be sweet, though difficult, if you didn't have to quiesce hbase
making a backup. See https://issues.apache.org/jira/browse/HBASE-50
for some kicking around of ideas.
St.Ack
