This is a follow up of what I have found . I exported the several
complained tables to hdfs, truncate the original table, and import it again,
and run hbck, and found that the hbck still complain the problem saying the
hdfs directory is not there. I go to hdfs and take a look, and the region's
hdfs directory is there. so the hbck's complain is bogus this time.
By accident, I run the same hbck on one of the regionserver, and to my
surprise, the hbck check comes out clean for all tables ! I then run this
command in several other regionserver, and then all 3 hbase masters, all of
the come out clean ,
even for the table that has problem before and I didn't export and import.
I tried several other non-hbase machines that has proper configuration, sure
enough, all of them complain problems.
So it seems the result of hbck depends on non-hbase machine or hbase
machine. Judging from the results they show,
none of them is correct. The correct result should be the imported tables
are clean and non-imported tables are not.
Can anybody explain why hbck have this kind of behavior ?
Jimmy
--------------------------------------------------
From: "Jinsong Hu" <[email protected]>
Sent: Monday, May 23, 2011 11:39 AM
To: <[email protected]>
Subject: Re: hbase hbck error
I checked the master, unfortunately , I must have wrong setting that all
master log are not there.
So I checked the regionserver which hosted this region. I have 14 days
log there and I grep this 02f9ec575b19864ae44e714d9245138f,
and I don't see any log. then I searched all regionserver's log for last
several days , and don't see
any log related to this region either.
Jimmy.
--------------------------------------------------
From: "Jean-Daniel Cryans" <[email protected]>
Sent: Monday, May 23, 2011 10:53 AM
To: <[email protected]>
Subject: Re: hbase hbck error
I don't remember seeing this sort of issue a lot, or at all... Usually
the region would not be on .META. so it looks like a different issue.
Could you grep the master logs and see what's the story of that
region? Just look for 02f9ec575b19864ae44e714d9245138f and try to
figure what happened to that region, might give us a clue.
J-D
On Mon, May 23, 2011 at 10:29 AM, Jinsong Hu <[email protected]>
wrote:
Hi,
today I run "hbase hbck " to check our production cluster and dev
cluster,
the production cluster comes out clean, but
in our dev cluster, I have seem more than 2K errors like this:
ERROR: Region
HEARTBEAT_MASTERPATCH,time\x09daily\x092010-08-15\x09uobkayhian_pr
oduction\x09patch-0000694,1287356584131.02f9ec575b19864ae44e714d9245138f.
found
in META, but not in HDFS, and deployed on m0002040.ppops.net:60020
I checked hbase GUI, and indeed , it is correct, the region is loaded by
the
region server, but the hdfs directory
is not there.
I am running cdh3u0, and I wonder how this can happen. Once it has
happened,
what can I do to recover to bring the table back to healthy state.
Jimmy.
