We had an issue a day ago with some OOME's on the region servers. The
master shutdown ok, but most of the RegionServers didn't and so eventually
had to kill -9 them. Brought it all back up and ran a major compaction to
change the hbase block size. This seemed to work, but now we have an
inconsistency which is preventing bulk loads from continuing.
hbase hbck -details finds an inconsistency. I tried -fix, but no help.
*Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
*
* *
hbck also notes that this region is offline:
*11/03/05 23:33:16 DEBUG util.HBaseFsck: Region
opx_ad_event_v2,advertiser\x02OpenX
PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
x02thejesperbay.com\x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600,1297185243218.6147d3696ba9db3a85e3afd08d0bc59a.
offline, split, parent, ignoring.*
Looking in .META. I see that the region is indeed offline, and appears to be
split:
info:regioninfo timestamp=1299301154675 ...
OFFLINE => true,
info:splitA timestamp=1299283401019
info:splitB timestamp=1299283401019
(full .META. row below)
So, I'm guessing that it was in the midst of splitting and did not complete.
How can I recover from this situation?
thanks,
Marc
----------- .META. output ----------------
hbase(main):001:0> get '.META.' , "opx_ad_event_v2,advertiser\x02OpenX
PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
x02thejesperbay.com\x01advert""
COLUMN
CELL
info:regioninfo timestamp=1299301154675,
value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
PSA\x01country\x02Serbia\x01publisher\x02DS
NR - Filesharing
ROW\x01domain\x02thejesperbay.com\x01advertiser_tag\x02mmx.travel
\x01starttime\x021295175600,1297185243218.6147d3
696ba9db3a85e3afd08d0bc59a.',
STARTKEY => 'advertiser\x02OpenX
PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01
domain\x02thejesperbay.com
\x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600', ENDKEY =>
'advertiser\x02OpenX PSA\x01coun
try\x02United Arab
Emirates\x01publisher\
x02www.sixbillionsecrets.com/\x01advertiser_tag\x02mmx.arts and
entertainment\x01publishe
r_tag\x02mmx.arts and
entertainment\x01starttime\x021295877600', ENCODED =>
6147d3696ba9db3a85e3afd08d0bc59a, OFFLINE => true, SPL
IT => true, TABLE => {{NAME =>
'opx_ad_event_v2', FAMILIES => [{NAME => 'metrics', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0'
, VERSIONS => '1', COMPRESSION
=> 'GZ', TTL => '2147483647', BLOCKSIZE => '1048576', IN_MEMORY => 'false',
BLOCKCACHE => 'true'},
{NAME => 'topn', BLOOMFILTER
=> 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'GZ', VERSIONS => '1',
TTL => '2147483647', BLOCK
SIZE => '1048576', IN_MEMORY
=> 'false', BLOCKCACHE =>
'true'}]}}
info:server timestamp=1299181144063,
value=ip-10-17-24-121.ec2.internal:60020
info:serverstartcode timestamp=1299181144063,
value=1299180905510
info:splitA timestamp=1299283401019,
value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
PSA\x01country\x02Serbia\x01publisher\x02DS
NR - Filesharing
ROW\x01domain\x02thejesperbay.com\x01advertiser_tag\x02mmx.travel
\x01starttime\x021295175600,1299283399612.3b278f
1b0ea78af239409efc4f0b2a3d.',
STARTKEY => 'advertiser\x02OpenX
PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01
domain\x02thejesperbay.com
\x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600', ENDKEY =>
'advertiser\x02OpenX PSA\x01coun
try\x02Taiwan\x01domain\
x02kanzhongguo.com\x01advertiser_tag\x02mmx.arts and
entertainment\x01publisher_tag\x02\x01starttime\x0212
96910800', ENCODED =>
3b278f1b0ea78af239409efc4f0b2a3d, TABLE => {{NAME => 'opx_ad_event_v2',
FAMILIES => [{NAME => 'metrics', BLO
OMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'GZ', TTL =>
'2147483647', BLOCKSIZE => '65536', IN_
MEMORY => 'false', BLOCKCACHE
=> 'true'}, {NAME => 'topn', BLOOMFILTER => 'NONE', REPLICATION_SCOPE =>
'0', VERSIONS => '1', COMPR
ESSION => 'GZ', TTL =>
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}]}}
info:splitB timestamp=1299283401019,
value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
PSA\x01country\x02Taiwan\x01domain\x02kanzh
ongguo.com\x01advertiser_tag\x02mmx.arts
and
entertainment\x01publisher_tag\x02\x01starttime\x021296910800,1299283399612.9d4164620
2a363812d068792311d3a9b.',
STARTKEY => 'advertiser\x02OpenX PSA\x01country\x02Taiwan\x01domain\
x02kanzhongguo.com\x01advertiser_ta
g\x02mmx.arts and
entertainment\x01publisher_tag\x02\x01starttime\x021296910800', ENDKEY =>
'advertiser\x02OpenX PSA\x01country\x0
2United Arab
Emirates\x01publisher\
x02www.sixbillionsecrets.com/\x01advertiser_tag\x02mmx.arts and
entertainment\x01publisher_tag\
x02mmx.arts and
entertainment\x01starttime\x021295877600', ENCODED =>
9d41646202a363812d068792311d3a9b, TABLE => {{NAME => 'opx_ad
_event_v2', FAMILIES => [{NAME
=> 'metrics', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS =>
'1', COMPRESSION => 'GZ'
, TTL => '2147483647',
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME =>
'topn', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0',
VERSIONS => '1', COMPRESSION => 'GZ', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', B
LOCKCACHE =>
'true'}]}}
5 row(s) in 0.3960 seconds
------------ hbck -details output -------------
...
ERROR: Region
hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/34a0ffe60da97431a809f0ffe8e5328a
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/3b278f1b0ea78af239409efc4f0b2a3d
on HDFS, but not listed in META or deployed on any region server.
*11/03/05 23:33:16 DEBUG util.HBaseFsck: Region
opx_ad_event_v2,advertiser\x02OpenX
PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
x02thejesperbay.com\x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600,1297185243218.6147d3696ba9db3a85e3afd08d0bc59a.
offline, split, parent, ignoring.
*ERROR: Region
hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/9d41646202a363812d068792311d3a9b
on HDFS, but not listed in META or deployed on any region server.
*Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
E*RROR: Found inconsistency in table opx_ad_event_v2
Summary:
-ROOT- is okay.
Number of regions: 1
Deployed on: ip-10-17-24-121.ec2.internal:60020
.META. is okay.
Number of regions: 1
Deployed on: ip-10-17-5-252.ec2.internal:60020
...
Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
Table opx_ad_event_v2 is inconsistent.
Number of regions: 1612
Deployed on: ...
4 inconsistencies detected.
Status: INCONSISTENT
