We also check the master log, nothing interesting found.
On Wed, Jul 6, 2011 at 12:58 AM, Xu-Feng Mao <m9s...@gmail.com> wrote:
> Hi,
>
> We're running a hbase cluster including 37 regionservers. Today, we found
> losts of WrongRegionException when putting object into it.
>
> hbase hbck -details
> reports that
> ====
> Chain of regions in table STable is broken; edges does not contain
> ztxrGmCwn-6BE32s3cX1TNeHU_I=
> ERROR: Found inconsistency in table STable
> ====
>
> echo "scan '.META.'"| hbase shell &> meta.txt
> grep -A1 "STARTKEY => 'EStore_everbox_z" meta.txt
> reports that
> ====
> Ck=,1308802977279.71ffb1 1ffb10b8b95fd47b3eff468d00ab4e9.',
> STARTKEY => 'ztn0ukLW
> 0b8b95fd47b3eff468d00ab4 d1NSU3fuXKkkWq5ZVCk=', ENDKEY =>
> 'ztqdVD8fCMP-dDbXUAydan
> e9. kboD4=', ENCODED =>
> 71ffb10b8b95fd47b3eff468d00ab4e9, TABLE => {{NAME =
> --
> D4=,1305619724446.c45191 45191821053d03537596f4a2e759718.',
> STARTKEY => ztqdVD8f
> 821053d03537596f4a2e7597 CMP-dDbXUAydankboD4=', ENDKEY => '
> ztxrGmCwn-6BE32s3cX1TN
> 18. eHU_I=', ENCODED =>
> c45191821053d03537596f4a2e759718, TABLE => {{NAME =
> --
> pA=,1309455605341.c5c5f5 5c5f578722ea3f8d1b099313bec8298.',
> STARTKEY => 'zu3zVaLc
> 78722ea3f8d1b099313bec82 GDnnpjKCbnboXgAFspA=', ENDKEY =>
> 'zu7qkr5fH6MMJ3GxbCv_0d
> 98. 6g8yI=', ENCODED =>
> c5c5f578722ea3f8d1b099313bec8298, TABLE => {{NAME =
> ====
>
> It looks like the meta indeed has a hole.(We tried scan '.META.' several
> times, to confirm it's not a transient status.)
> We've tried hbase hbck -fix, does not help.
>
> We found a thread 'wrong region exception' about two months ago. Stack
> suggested a 'little surgery' like
> ====
>
> *So, make sure you actually have a hole. Dump out your meta table:
>
> echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt
>
> Then look ensure that there is a hole between the above regions
> (compare start and end keys... the end key of one region needs to
> match the start key of the next).
>
> If indeed a hole, you need to do a little surgery inserting a new
> missing region (hbck should fix this but it doesn't have the smarts
> just yet).
>
> Basically, you create a new region with start and end keys to fill the
> hole then you insert it into .META. and then assign it. There are
> some scripts in our bin directory that do various parts of this. I'm
> pretty sure its beyond any but a few figuring this mess out so if you
> do the above foot work and provide a few more details, I'll hack up
> something for you (and hopefully something generalized to be use by
> others later, and later to be integrated into hbck).*
>
> ====
>
> Can anyone give a detailed example, step by step instruction would be
> greatly appreciated.
> My understand is we should
> 1.Since we already has the lost region, we now have start and end keys.
> 2.generate the row represents the missing region. But how can I generate
> the encoded name?
> It looks like I need
> column=info:server,column=info:serverstartcode and column=info:regioninfo
> for the missing region.
> And column=info:regioninfo includes so many information. How to generate
> them one by one?
> As for the name of row, it consists of tablename, startkey, encode, and one
> more long number,
> how to get this number?
> 3.use assing command in the hbase shell
>
> We also tried check_meta.rb --fix, it reports
> ====
> 11/07/06 00:09:08 WARN check_meta: hole after REGION => {NAME =>
> 'STable,ztqdVD8fCMP-dDbXUAydankboD4=,1305619724446.c45191821053d03537596f4a2e759718.',
> STARTKEY => 'ztqdVD8fCMP-dDbXUAydankboD4=', ENDKEY =>
> 'ztxrGmCwn-6BE32s3cX1TNeHU_I=', ENCODED => c45191821053d03537596f4a2e759718,
> TABLE => {{NAME => 'STable', FAMILIES => [{NAME => 'file', BLOOMFILTER =>
> 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3',
> TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'}, {NAME => 'filelength', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '
> 2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'true'}, {NAME => 'userbucket', BLOOMFILTER => 'NONE', REPLICATION_SCOPE =>
> '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME =>
> 'userpass', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION =>
> 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
> 11/07/06 00:28:40 WARN check_meta: Missing .regioninfo: hdfs://
> hd0013.c.gj.com:9000/hbase/STable/3e6faca40a7ccad7ed8c0b5848c0f945/.regioninfo
> ====
>
> The problem is still there. BTW, what about the blue warning? Is this a
> serious issue?
> The situation is quite hard to us, it looks like even we can fill the hole
> in the meta, we would lost all the data in the hole region, right?
>
> Thanks and regards,
>
> Mao Xu-Feng
>
