Yeah, I was the one that ran into the same problem. Took the check meta script
and converted it based on some of Stack's suggestions and our own experiments.
Also fixed boundary conditions for regions at the beginning and at the end of
the chain. In addition, we added the assign in here as well.
Here is the code
===========================BEGIN CODE================================
# Script looks at hbase .META. table verifying its content is coherent.
#
# To see usage for this script, run:
#
# ${HBASE_HOME}/bin/hbase org.jruby.Main check_meta.rb --help
#
# Copyright 2010 The Apache Software Foundation
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
include Java
import org.apache.commons.logging.LogFactory
import org.apache.hadoop.hbase.util.VersionInfo
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
import org.apache.hadoop.hbase.HConstants
import org.apache.hadoop.hbase.util.FSUtils
import org.apache.hadoop.hbase.client.HTable
import org.apache.hadoop.hbase.client.Scan
import org.apache.hadoop.hbase.client.HBaseAdmin
import org.apache.hadoop.hbase.util.Writables
import org.apache.hadoop.hbase.HRegionInfo
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.HTableDescriptor
import org.apache.hadoop.hbase.client.Put
# Name of this script
NAME = 'fixholes'
# Print usage for this script
def usage
puts 'Usage: %s.rb [--fix]' % NAME
puts ' fix Try to fixup meta issues including holes'
puts 'Script checks consistency of the .META. table. It reports if .META.
has missing entries.'
puts 'If you pass "--fix", it will re-create dropped region by pluggin the
.META. hole'
puts 'and assigning it, which will create a blank region file and assign it
to a server'
exit!
def isFixup
# Are we to do fixup during this run
usage if ARGV.size > 1
fixup = nil
if ARGV.size == 1
usage unless ARGV[0].downcase.match('--fix.*')
fixup = 1
end
return fixup
end
def getConfiguration
hbase_twenty = VersionInfo.getVersion().match('0\.20\..*')
# Get configuration to use.
if hbase_twenty
c = HBaseConfiguration.new()
else
c = HBaseConfiguration.create()
end
# Set hadoop filesystem configuration using the hbase.rootdir.
# Otherwise, we'll always use localhost though the hbase.rootdir
# might be pointing at hdfs location. Do old and new key for fs.
c.set("fs.default.name", c.get(HConstants::HBASE_DIR))
c.set("fs.defaultFS", c.get(HConstants::HBASE_DIR))
return c
end
def fixup(leftEdge, rightEdge, metatable, conf)
reBytes = nil
leBytes = nil
tabDesc = nil
if not leftEdge == ""
leBytes = leftEdge.getEndKey()
else
leBytes = Bytes.toBytes("")
tabDesc = rightEdge.getTableDesc();
end
if not rightEdge == ""
reBytes = rightEdge.getStartKey()
tabDesc = rightEdge.getTableDesc();
else
reBytes = Bytes.toBytes("")
tabDesc = leftEdge.getTableDesc();
end
hri = HRegionInfo.new(tabDesc, leBytes, reBytes)
p = Put.new(hri.getRegionName())
p.add(HConstants::CATALOG_FAMILY, HConstants::REGIONINFO_QUALIFIER,
Writables.getBytes(hri))
metatable.put(p)
LOG.info("Plugged hole in .META. at: " + hri.toString())
hba = HBaseAdmin.new(conf);
hba.assign(hri.getRegionName(), true);
LOG.info("Assigned new region to plugged hole");
return true
end
fixup = isFixup()
# Get configuration
conf = getConfiguration()
# Filesystem
fs = FileSystem.get(conf)
# Rootdir
rootdir = FSUtils.getRootDir(conf)
# Get a logger and a metautils instance.
LOG = LogFactory.getLog(NAME)
# Scan the .META. looking for holes
metatable = HTable.new(conf, HConstants::META_TABLE_NAME)
scan = Scan.new()
scanner = metatable.getScanner(scan)
oldHRI = nil
bad = 0
while (result = scanner.next())
rowid = Bytes.toString(result.getRow())
rowidStr = java.lang.String.new(rowid)
bytes = result.getValue(HConstants::CATALOG_FAMILY,
HConstants::REGIONINFO_QUALIFIER)
next if not bytes
next if bytes.length == 0
hri = Writables.getHRegionInfo(bytes)
if oldHRI
if oldHRI.isOffline() && Bytes.equals(oldHRI.getStartKey(),
hri.getStartKey())
# Presume offlined parent
elsif Bytes.equals(oldHRI.getEndKey(), hri.getStartKey())
# Start key of next matches end key of previous
else
LOG.info("hole after " + oldHRI.toString())
if fixup
bad = 1 unless fixup(oldHRI, hri, metatable, conf)
else
bad = 1
end
end
else
if not Bytes.toString(hri.getStartKey()) == ""
bad = 1 unless fixup("", hri, metatable, conf)
end
end
oldHRI = hri
end
if not Bytes.toString(hri.getEndKey()) == ""
bad = 1 unless fixup(hri, "", metatable, conf)
end
scanner.close()
if bad
LOG.info(".META. has holes")
else
LOG.info(".META. is healthy")
end
# Return 0 if meta is good, else non-zero.
exit bad
===========================END CODE================================
-----Original Message-----
From: Xu-Feng Mao [mailto:[email protected]]
Sent: Tuesday, July 05, 2011 6:21 PM
To: Xu-Feng Mao
Cc: [email protected]; [email protected]
Subject: Re: WrongRegionException and inconsistent table found
I forgot the version, we are using cdh3u0.
Mao Xu-Feng
在 2011-7-6，0:59，Xu-Feng Mao <[email protected]> 写道：
We also check the master log, nothing interesting found.
On Wed, Jul 6, 2011 at 12:58 AM, Xu-Feng Mao <[email protected]> wrote:
> Hi,
>
> We're running a hbase cluster including 37 regionservers. Today, we
> found losts of WrongRegionException when putting object into it.
>
> hbase hbck -details
> reports that
> ====
> Chain of regions in table STable is broken; edges does not contain
> ztxrGmCwn-6BE32s3cX1TNeHU_I=
> ERROR: Found inconsistency in table STable ====
>
> echo "scan '.META.'"| hbase shell &> meta.txt grep -A1 "STARTKEY =>
> 'EStore_everbox_z" meta.txt
> reports that
> ====
> Ck=,1308802977279.71ffb1 1ffb10b8b95fd47b3eff468d00ab4e9.',
> STARTKEY => 'ztn0ukLW
> 0b8b95fd47b3eff468d00ab4 d1NSU3fuXKkkWq5ZVCk=', ENDKEY =>
> 'ztqdVD8fCMP-dDbXUAydan
> e9. kboD4=', ENCODED =>
> 71ffb10b8b95fd47b3eff468d00ab4e9, TABLE => {{NAME =
> --
> D4=,1305619724446.c45191 45191821053d03537596f4a2e759718.',
> STARTKEY => ztqdVD8f
> 821053d03537596f4a2e7597 CMP-dDbXUAydankboD4=', ENDKEY => '
> ztxrGmCwn-6BE32s3cX1TN
> 18. eHU_I=', ENCODED =>
> c45191821053d03537596f4a2e759718, TABLE => {{NAME =
> --
> pA=,1309455605341.c5c5f5 5c5f578722ea3f8d1b099313bec8298.',
> STARTKEY => 'zu3zVaLc
> 78722ea3f8d1b099313bec82 GDnnpjKCbnboXgAFspA=', ENDKEY =>
> 'zu7qkr5fH6MMJ3GxbCv_0d
> 98. 6g8yI=', ENCODED =>
> c5c5f578722ea3f8d1b099313bec8298, TABLE => {{NAME = ====
>
> It looks like the meta indeed has a hole.(We tried scan '.META.'
> several times, to confirm it's not a transient status.) We've tried
> hbase hbck -fix, does not help.
>
> We found a thread 'wrong region exception' about two months ago. Stack
> suggested a 'little surgery' like ====
>
> *So, make sure you actually have a hole. Dump out your meta table:
>
> echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt
>
> Then look ensure that there is a hole between the above regions
> (compare start and end keys... the end key of one region needs to
> match the start key of the next).
>
> If indeed a hole, you need to do a little surgery inserting a new
> missing region (hbck should fix this but it doesn't have the smarts
> just yet).
>
> Basically, you create a new region with start and end keys to fill the
> hole then you insert it into .META. and then assign it. There are
> some scripts in our bin directory that do various parts of this. I'm
> pretty sure its beyond any but a few figuring this mess out so if you
> do the above foot work and provide a few more details, I'll hack up
> something for you (and hopefully something generalized to be use by
> others later, and later to be integrated into hbck).*
>
> ====
>
> Can anyone give a detailed example, step by step instruction would be
> greatly appreciated.
> My understand is we should
> 1.Since we already has the lost region, we now have start and end keys.
> 2.generate the row represents the missing region. But how can I
> generate the encoded name?
> It looks like I need
> column=info:server,column=info:serverstartcode and
> column=info:regioninfo for the missing region.
> And column=info:regioninfo includes so many information. How to
> generate them one by one?
> As for the name of row, it consists of tablename, startkey, encode,
> and one more long number, how to get this number?
> 3.use assing command in the hbase shell
>
> We also tried check_meta.rb --fix, it reports ====
> 11/07/06 00:09:08 WARN check_meta: hole after REGION => {NAME =>
> 'STable,ztqdVD8fCMP-dDbXUAydankboD4=,1305619724446.c45191821053d035375
> 96f4a2e759718.', STARTKEY => 'ztqdVD8fCMP-dDbXUAydankboD4=', ENDKEY =>
> 'ztxrGmCwn-6BE32s3cX1TNeHU_I=', ENCODED =>
> c45191821053d03537596f4a2e759718, TABLE => {{NAME => 'STable',
> FAMILIES => [{NAME => 'file', BLOOMFILTER => 'NONE', REPLICATION_SCOPE
> => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
> {NAME => 'filelength', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
> COMPRESSION => 'NONE', VERSIONS => '3', TTL => '
> 2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'true'}, {NAME => 'userbucket', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL
> => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'}, {NAME => 'userpass', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL
> => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'}]}}
> 11/07/06 00:28:40 WARN check_meta: Missing .regioninfo: hdfs://
> hd0013.c.gj.com:9000/hbase/STable/3e6faca40a7ccad7ed8c0b5848c0f945/.re
> gioninfo
> ====
>
> The problem is still there. BTW, what about the blue warning? Is this
> a serious issue?
> The situation is quite hard to us, it looks like even we can fill the
> hole in the meta, we would lost all the data in the hole region, right?
>
> Thanks and regards,
>
> Mao Xu-Feng
>
