The related log is at: http://pastebin.com/0a1CjDUD
It's ok now after restarting hbase, but still curious why it happend. Thanks, Yi On Sat, Feb 19, 2011 at 3:58 AM, Jean-Daniel Cryans <[email protected]>wrote: > The master should finish processing those dead servers at some point > and it seems it's not happening? Unfortunately without the log nobody > can'tell why. If you can post the complete log in pastebin or put it > on a web server then we could take a look. > > J-D > > On Fri, Feb 18, 2011 at 12:39 AM, Yi Liang <[email protected]> wrote: > > Hi all, > > > > We have a hbase cluster with 10 region servers running HBase 0.90.0 + > CDH3. > > We're now importing big data into HBase. > > > > During the process, 2 servers crashed, but after restaring them, they're > no > > longer assigned with any region, while regions on other servers keep > > splitting when more data inserted. > > > > From the master log, we can see the periodical messages like: > > > > 2011-02-18 16:09:35,067 DEBUG org.apache.hadoop.hbase.master.HMaster: Not > > running balancer because processing dead regionserver(s): > > [zcl.local,60020,1297996817352, qics.local,60020,1297919358488, > > Docete.local,60020,1297919410096, liym.local,60020,1297919445796, > > zcl.local,60020,1297919367472] > > > > zcl.local and qics.local are the machines we have restared, other 2 > machine > > have kept running without restarting and are actually still serving > regions. > > > > From the shell status: > > 10 servers, 5 dead, 10.1000 average Load > > > > Why are there dead servers? And how to clear them so we could start > > balancer? > > > > Thanks, > > Yi > > >
