I updated trunk and copied it to all the machines just before I
started the import.
I will try and replicate the problem again.
>scan '.META.'
gives this:
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.2.0, r682496, Tue Aug 5 11:02:10 BST 2008
hbase(main):001:0> scan '.META.'
08/08/06 10:39:12 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/06 10:39:12 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/06 10:39:12 DEBUG client.HTable$ClientScanner: Creating scanner
over .META. starting at key ''
08/08/06 10:39:12 DEBUG client.HTable$ClientScanner: Advancing
internal scanner to startKey at ''
08/08/06 10:39:12 DEBUG client.HConnectionManager$TableServers: Cache
hit in table locations for row <> and tableName .META.: location
server 10.101.1.32:60020, location region name .META.,,1
ROW COLUMN+CELL
catalogue,,1217934372881 column=historian:assignment,
timestamp=1217952169102, value=Region assigned to se
rver 10.101.1.31:60020
catalogue,,1217934372881 column=historian:compaction,
timestamp=1217952169766, value=Region compaction com
pleted in 0sec
catalogue,,1217934372881 column=historian:open,
timestamp=1217952172128, value=Region opened on server : h
adoopdev1
catalogue,,1217934372881 column=historian:split,
timestamp=1217934343125, value=Region split from : catal
ogue,,1217518786594
catalogue,,1217934372881 column=info:regioninfo,
timestamp=1217934343130, value=REGION => {NAME => 'catalo
gue,,1217934372881', STARTKEY => '',
ENDKEY => '20039453', ENCODED => 269352862,
TABLE => {{NAME => 'catalogue', IS_ROOT
=> 'false', IS_META => 'false', FAMILIES
=> [{NAME => 'track', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS =>
'3', LENGTH => '2147483647', TTL => '-1',
IN_MEMORY => 'false', BLOCKCACHE => 'fa
lse'}, {NAME => 'track_permissions',
BLOOMFILTER => 'false', COMPRESSION => 'NONE
', VERSIONS => '3', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BL
OCKCACHE => 'false'}, {NAME =>
'album_track_join', BLOOMFILTER => 'false', COMPRE
SSION => 'NONE', VERSIONS => '3', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY
=> 'false', BLOCKCACHE => 'false'}, {NAME
=> 'album', BLOOMFILTER => 'false', COM
PRESSION => 'NONE', VERSIONS => '3',
LENGTH => '2147483647', TTL => '-1', IN_MEMO
RY => 'false', BLOCKCACHE => 'false'}]}}
catalogue,,1217934372881 column=info:server,
timestamp=1217952172126, value=10.101.1.31:60020
catalogue,,1217934372881 column=info:serverstartcode,
timestamp=1217952172126, value=1217952136835
catalogue,20039453,12179343 column=historian:assignment,
timestamp=1217952169104, value=Region assigned to se
72881 rver 10.101.1.31:60020
catalogue,20039453,12179343 column=historian:compaction,
timestamp=1217952169966, value=Region compaction com
72881 pleted in 0sec
catalogue,20039453,12179343 column=historian:open,
timestamp=1217952172133, value=Region opened on server : h
72881 adoopdev1
catalogue,20039453,12179343 column=historian:split,
timestamp=1217934343127, value=Region split from : catal
72881 ogue,,1217518786594
catalogue,20039453,12179343 column=info:regioninfo,
timestamp=1217934343131, value=REGION => {NAME => 'catalo
72881 gue,20039453,1217934372881', STARTKEY =>
'20039453', ENDKEY => '', ENCODED => 956
258381, TABLE => {{NAME => 'catalogue',
IS_ROOT => 'false', IS_META => 'false', F
AMILIES => [{NAME => 'track', BLOOMFILTER
=> 'false', COMPRESSION => 'NONE', VERS
IONS => '3', LENGTH => '2147483647', TTL
=> '-1', IN_MEMORY => 'false', BLOCKCACH
E => 'false'}, {NAME =>
'track_permissions', BLOOMFILTER => 'false', COMPRESSION
=> 'NONE', VERSIONS => '3', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'fa
lse', BLOCKCACHE => 'false'}, {NAME =>
'album_track_join', BLOOMFILTER => 'false'
, COMPRESSION => 'NONE', VERSIONS => '3',
LENGTH => '2147483647', TTL => '-1', IN
_MEMORY => 'false', BLOCKCACHE =>
'false'}, {NAME => 'album', BLOOMFILTER => 'fal
se', COMPRESSION => 'NONE', VERSIONS =>
'3', LENGTH => '2147483647', TTL => '-1',
IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
catalogue,20039453,12179343 column=info:server,
timestamp=1217952172131, value=10.101.1.31:60020
72881
catalogue,20039453,12179343 column=info:serverstartcode,
timestamp=1217952172131, value=1217952136835
72881
catalogue_album,,1216910554 column=historian:assignment,
timestamp=1217952174659, value=Region assigned to se
819 rver 10.101.1.33:60020
catalogue_album,,1216910554 column=historian:compaction,
timestamp=1217952175333, value=Region compaction com
819 pleted in 0sec
catalogue_album,,1216910554 column=historian:creation,
timestamp=1216910554877, value=Region creation
819
catalogue_album,,1216910554 column=historian:open,
timestamp=1217952177679, value=Region opened on server : h
819 adoopdev3.bra.int.last.fm
catalogue_album,,1216910554 column=info:regioninfo,
timestamp=1216910554911, value=REGION => {NAME => 'catalo
819 gue_album,,1216910554819', STARTKEY =>
'', ENDKEY => '', ENCODED => 1294877719, T
ABLE => {{NAME => 'catalogue_album',
IS_ROOT => 'false', IS_META => 'false', FAMI
LIES => [{NAME => 'entry', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSION
S => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE =
> 'false'}]}}
catalogue_album,,1216910554 column=info:server,
timestamp=1217952177676, value=10.101.1.33:60020
819
catalogue_album,,1216910554 column=info:serverstartcode,
timestamp=1217952177676, value=1217952167273
819
catalogue_track,,1216910536 column=historian:assignment,
timestamp=1217952174657, value=Region assigned to se
829 rver 10.101.1.33:60020
catalogue_track,,1216910536 column=historian:compaction,
timestamp=1217952174933, value=Region compaction com
829 pleted in 0sec
catalogue_track,,1216910536 column=historian:creation,
timestamp=1216910536901, value=Region creation
829
catalogue_track,,1216910536 column=historian:open,
timestamp=1217952177674, value=Region opened on server : h
829 adoopdev3.bra.int.last.fm
catalogue_track,,1216910536 column=info:regioninfo,
timestamp=1216910536941, value=REGION => {NAME => 'catalo
829 gue_track,,1216910536829', STARTKEY =>
'', ENDKEY => '', ENCODED => 1548398197, T
ABLE => {{NAME => 'catalogue_track',
IS_ROOT => 'false', IS_META => 'false', FAMI
LIES => [{NAME => 'entry', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSION
S => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE =
> 'false'}]}}
catalogue_track,,1216910536 column=info:server,
timestamp=1217952177670, value=10.101.1.33:60020
829
catalogue_track,,1216910536 column=info:serverstartcode,
timestamp=1217952177670, value=1217952167273
829
08/08/06 10:39:13 DEBUG client.HTable$ClientScanner: Advancing forward
from region REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY =>
'', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER
=> 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS =>
'2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
28 row(s) in 0.4080 seconds
2008/8/5 stack <[EMAIL PROTECTED]>:
> Tim Sell wrote:
>>
>> ...
>>
>> I understand until recently the logs were filled with warnings about
>> this during every split while a region was temporarily offline so to
>> speak? it would figure itself out once the region was opened, but
>> logging these warnings was removed because it was confusing people?
>>
>>
>
> No. They are still there. They are logged at DEBUG level unless we exhaust
> retries. Then you'll get a NSRE out on your client as an error (That said,
> they are showing in your logs here because they are logged at INFO level by
> hadoop ipc).
>
>> This isn't one of those warnings, it's the only
>> NotServingRegionException in my logs.
>> I'm using this morning's trunk.
>>
>> I've attached my master and region server logs, if anyone can help. It
>> happens around the 2008-08-05 12:06 mark.
>>
>> I added some blank lines to help you find where the activity stopped.
>> (there is about a 30 minute gap in the region server logs afterwards
>> as there were no more input to be compacted, split, etc).
>>
>>
>
> I took a look. Seems fine.
>
> As Andrew asks, the thrift client log would be handy....
>
> But this in your second message is disturbing:
>
> 08/08/05 13:56:38 DEBUG client.HConnectionManager$TableServers:
> reloading table servers because: HRegionInfo was null or empty in
> .META.
>
> We're stuck trying to get .META. info but some rows are damaged; have null
> content.
>
> For sure you are at TRUNK from this morning? I thought we printed out all
> that was non-empty from the row to give us a clue which row is busted.
>
> What happens if you try and scan using the shell?
>
>> scan '.META.'
>
> St.Ack
>
>
>> [Towards the end there is some irrelevant stuff logged because I was
>> fiddling in the hbase shell. ignore that]
>>
>> ~Tim.
>>
>>
>> 2008/8/5 Andrew Purtell <[EMAIL PROTECTED]>:
>>
>>>
>>> Hello Nickolai,
>>>
>>> When your regions unexpectedly closed, is/was there anything
>>> logged to the regionserver and master logs at those times?
>>>
>>> As for your problem right now, please consider stopping, moving
>>> the old log files out of the way, and then restarting. Please
>>> then try to access your table. Then, please send the resulting
>>> master and region server logs. I'll be happy to take a look.
>>>
>>> Thanks,
>>>
>>> - Andy
>>>
>>>
>>>>
>>>> From: Nickolai Toupikov <[EMAIL PROTECTED]>
>>>> Subject: Issue with NotServingRegionException
>>>> To: [email protected]
>>>> Date: Tuesday, August 5, 2008, 5:47 AM
>>>> Hello!
>>>> It seems I have a serious problem with HBase. We've
>>>> been using if for some time now, had some minor problems with
>>>> regions unexpectedly closing once every couple of days, but
>>>> this would usually sort itself out after a couple of stops and
>>>> starts.
>>>>
>>>
>>>
>>>
>>
>> >
>
>
