I updated trunk and copied it to all the machines just before I
started the import.
I will try and replicate the problem again.
scan '.META.'
gives this:
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.2.0, r682496, Tue Aug 5 11:02:10 BST 2008
hbase(main):001:0> scan '.META.'
08/08/06 10:39:12 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/06 10:39:12 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/06 10:39:12 DEBUG client.HTable$ClientScanner: Creating scanner
over .META. starting at key ''
08/08/06 10:39:12 DEBUG client.HTable$ClientScanner: Advancing
internal scanner to startKey at ''
08/08/06 10:39:12 DEBUG client.HConnectionManager$TableServers: Cache
hit in table locations for row <> and tableName .META.: location
server 10.101.1.32:60020, location region name .META.,,1
ROW COLUMN+CELL
catalogue,,1217934372881 column=historian:assignment,
timestamp=1217952169102, value=Region assigned to se
rver 10.101.1.31:60020
catalogue,,1217934372881 column=historian:compaction,
timestamp=1217952169766, value=Region compaction com
pleted in 0sec
catalogue,,1217934372881 column=historian:open,
timestamp=1217952172128, value=Region opened on server : h
adoopdev1
catalogue,,1217934372881 column=historian:split,
timestamp=1217934343125, value=Region split from : catal
ogue,,1217518786594
catalogue,,1217934372881 column=info:regioninfo,
timestamp=1217934343130, value=REGION => {NAME => 'catalo
gue,,1217934372881', STARTKEY => '',
ENDKEY => '20039453', ENCODED => 269352862,
TABLE => {{NAME => 'catalogue', IS_ROOT
=> 'false', IS_META => 'false', FAMILIES
=> [{NAME => 'track', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS =>
'3', LENGTH => '2147483647', TTL => '-1',
IN_MEMORY => 'false', BLOCKCACHE => 'fa
lse'}, {NAME => 'track_permissions',
BLOOMFILTER => 'false', COMPRESSION => 'NONE
', VERSIONS => '3', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BL
OCKCACHE => 'false'}, {NAME =>
'album_track_join', BLOOMFILTER => 'false', COMPRE
SSION => 'NONE', VERSIONS => '3', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY
=> 'false', BLOCKCACHE => 'false'}, {NAME
=> 'album', BLOOMFILTER => 'false', COM
PRESSION => 'NONE', VERSIONS => '3',
LENGTH => '2147483647', TTL => '-1', IN_MEMO
RY => 'false', BLOCKCACHE => 'false'}]}}
catalogue,,1217934372881 column=info:server,
timestamp=1217952172126, value=10.101.1.31:60020
catalogue,,1217934372881 column=info:serverstartcode,
timestamp=1217952172126, value=1217952136835
catalogue,20039453,12179343 column=historian:assignment,
timestamp=1217952169104, value=Region assigned to se
72881 rver 10.101.1.31:60020
catalogue,20039453,12179343 column=historian:compaction,
timestamp=1217952169966, value=Region compaction com
72881 pleted in 0sec
catalogue,20039453,12179343 column=historian:open,
timestamp=1217952172133, value=Region opened on server : h
72881 adoopdev1
catalogue,20039453,12179343 column=historian:split,
timestamp=1217934343127, value=Region split from : catal
72881 ogue,,1217518786594
catalogue,20039453,12179343 column=info:regioninfo,
timestamp=1217934343131, value=REGION => {NAME => 'catalo
72881 gue,20039453,1217934372881', STARTKEY =>
'20039453', ENDKEY => '', ENCODED => 956
258381, TABLE => {{NAME => 'catalogue',
IS_ROOT => 'false', IS_META => 'false', F
AMILIES => [{NAME => 'track', BLOOMFILTER
=> 'false', COMPRESSION => 'NONE', VERS
IONS => '3', LENGTH => '2147483647', TTL
=> '-1', IN_MEMORY => 'false', BLOCKCACH
E => 'false'}, {NAME =>
'track_permissions', BLOOMFILTER => 'false', COMPRESSION
=> 'NONE', VERSIONS => '3', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'fa
lse', BLOCKCACHE => 'false'}, {NAME =>
'album_track_join', BLOOMFILTER => 'false'
, COMPRESSION => 'NONE', VERSIONS => '3',
LENGTH => '2147483647', TTL => '-1', IN
_MEMORY => 'false', BLOCKCACHE =>
'false'}, {NAME => 'album', BLOOMFILTER => 'fal
se', COMPRESSION => 'NONE', VERSIONS =>
'3', LENGTH => '2147483647', TTL => '-1',
IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
catalogue,20039453,12179343 column=info:server,
timestamp=1217952172131, value=10.101.1.31:60020
72881
catalogue,20039453,12179343 column=info:serverstartcode,
timestamp=1217952172131, value=1217952136835
72881
catalogue_album,,1216910554 column=historian:assignment,
timestamp=1217952174659, value=Region assigned to se
819 rver 10.101.1.33:60020
catalogue_album,,1216910554 column=historian:compaction,
timestamp=1217952175333, value=Region compaction com
819 pleted in 0sec
catalogue_album,,1216910554 column=historian:creation,
timestamp=1216910554877, value=Region creation
819
catalogue_album,,1216910554 column=historian:open,
timestamp=1217952177679, value=Region opened on server : h
819 adoopdev3.bra.int.last.fm
catalogue_album,,1216910554 column=info:regioninfo,
timestamp=1216910554911, value=REGION => {NAME => 'catalo
819 gue_album,,1216910554819', STARTKEY =>
'', ENDKEY => '', ENCODED => 1294877719, T
ABLE => {{NAME => 'catalogue_album',
IS_ROOT => 'false', IS_META => 'false', FAMI
LIES => [{NAME => 'entry', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSION
S => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE =
> 'false'}]}}
catalogue_album,,1216910554 column=info:server,
timestamp=1217952177676, value=10.101.1.33:60020
819
catalogue_album,,1216910554 column=info:serverstartcode,
timestamp=1217952177676, value=1217952167273
819
catalogue_track,,1216910536 column=historian:assignment,
timestamp=1217952174657, value=Region assigned to se
829 rver 10.101.1.33:60020
catalogue_track,,1216910536 column=historian:compaction,
timestamp=1217952174933, value=Region compaction com
829 pleted in 0sec
catalogue_track,,1216910536 column=historian:creation,
timestamp=1216910536901, value=Region creation
829
catalogue_track,,1216910536 column=historian:open,
timestamp=1217952177674, value=Region opened on server : h
829 adoopdev3.bra.int.last.fm
catalogue_track,,1216910536 column=info:regioninfo,
timestamp=1216910536941, value=REGION => {NAME => 'catalo
829 gue_track,,1216910536829', STARTKEY =>
'', ENDKEY => '', ENCODED => 1548398197, T
ABLE => {{NAME => 'catalogue_track',
IS_ROOT => 'false', IS_META => 'false', FAMI
LIES => [{NAME => 'entry', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSION
S => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE =
> 'false'}]}}
catalogue_track,,1216910536 column=info:server,
timestamp=1217952177670, value=10.101.1.33:60020
829
catalogue_track,,1216910536 column=info:serverstartcode,
timestamp=1217952177670, value=1217952167273
829
08/08/06 10:39:13 DEBUG client.HTable$ClientScanner: Advancing forward
from region REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY =>
'', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER
=> 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS =>
'2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
28 row(s) in 0.4080 seconds
2008/8/5 stack <[EMAIL PROTECTED]>:
Tim Sell wrote:
...
I understand until recently the logs were filled with warnings about
this during every split while a region was temporarily offline so to
speak? it would figure itself out once the region was opened, but
logging these warnings was removed because it was confusing people?
No. They are still there. They are logged at DEBUG level unless we exhaust
retries. Then you'll get a NSRE out on your client as an error (That said,
they are showing in your logs here because they are logged at INFO level by
hadoop ipc).
This isn't one of those warnings, it's the only
NotServingRegionException in my logs.
I'm using this morning's trunk.
I've attached my master and region server logs, if anyone can help. It
happens around the 2008-08-05 12:06 mark.
I added some blank lines to help you find where the activity stopped.
(there is about a 30 minute gap in the region server logs afterwards
as there were no more input to be compacted, split, etc).
I took a look. Seems fine.
As Andrew asks, the thrift client log would be handy....
But this in your second message is disturbing:
08/08/05 13:56:38 DEBUG client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in
.META.
We're stuck trying to get .META. info but some rows are damaged; have null
content.
For sure you are at TRUNK from this morning? I thought we printed out all
that was non-empty from the row to give us a clue which row is busted.
What happens if you try and scan using the shell?
scan '.META.'
St.Ack
[Towards the end there is some irrelevant stuff logged because I was
fiddling in the hbase shell. ignore that]
~Tim.
2008/8/5 Andrew Purtell <[EMAIL PROTECTED]>:
Hello Nickolai,
When your regions unexpectedly closed, is/was there anything
logged to the regionserver and master logs at those times?
As for your problem right now, please consider stopping, moving
the old log files out of the way, and then restarting. Please
then try to access your table. Then, please send the resulting
master and region server logs. I'll be happy to take a look.
Thanks,
- Andy
From: Nickolai Toupikov <[EMAIL PROTECTED]>
Subject: Issue with NotServingRegionException
To: [email protected]
Date: Tuesday, August 5, 2008, 5:47 AM
Hello!
It seems I have a serious problem with HBase. We've
been using if for some time now, had some minor problems with
regions unexpectedly closing once every couple of days, but
this would usually sort itself out after a couple of stops and
starts.
