RE: HBase is not ready for Primetime

2011-04-13 Thread Doug Meil
@hbase.apache.org Cc: Robert Gonzalez Subject: Re: HBase is not ready for Primetime On Tue, Apr 12, 2011 at 10:52 AM, Robert Gonzalez g...@maxpointinteractive.com wrote: Ok, deleted logs that master was complaining about, restarted master only.  Seemed to be stable after a bunch of the messages like

RE: HBase is not ready for Primetime

2011-04-13 Thread Andrew Purtell
Hi Doug, 3) Cluster restart We schedule a full shutdown and restart of our cluster each week.  It's pretty quick, and HBase just seems happier when we do this. Can you say a bit more about how HBase is happier versus not? I can speculate on a number of reasons why this may be the case,

RE: HBase is not ready for Primetime

2011-04-13 Thread Doug Meil
. -Original Message- From: Andrew Purtell [mailto:apurt...@yahoo.com] Sent: Wednesday, April 13, 2011 1:51 PM To: user@hbase.apache.org Cc: Robert Gonzalez Subject: RE: HBase is not ready for Primetime Hi Doug, 3) Cluster restart We schedule a full shutdown and restart of our

Re: HBase is not ready for Primetime

2011-04-13 Thread Ryan Rawson
To bring it back to the original point and a high level view, the fact is that HBase is not Oracle, nor MySQL. It doesnt have multiple decades, and futhermore distributed systems are inherently more difficult (more failure cases) than single node DBs. Having said that, the grass is certainly not

Re: HBase is not ready for Primetime

2011-04-12 Thread Jinsong Hu
You probably should stop all master/regionservers, then start one master, tail -f the log to confirm all the hlogs are handled, then start the first regionserver, and then other regionservers. I have encountered this issues before. hbase is not as good as what you want, but not as bad as you

Re: HBase is not ready for Primetime

2011-04-12 Thread Bill Graham
Agreed. I've seen similar issues when upon startup where for whatever reason an hlog (often empty) can't be read, which hangs the startup process. Manually deleting it from HDFS clears the issue. On Tue, Apr 12, 2011 at 10:01 AM, Jinsong Hu jinsong...@hotmail.com wrote: You probably should stop

Re: HBase is not ready for Primetime

2011-04-12 Thread Robert Gonzalez
You mean like this: hbase@c1-m02:/usr/lib/hbase-0.90.0/bin$ ./stop-hbase.sh stopping

Re: HBase is not ready for Primetime

2011-04-12 Thread Ted Yu
In another terminal, you can see which region server(s) is still running. Checking region server log on that region server would help. On Tue, Apr 12, 2011 at 10:34 AM, Robert Gonzalez g...@maxpointinteractive.com wrote: You mean like this: hbase@c1-m02:/usr/lib/hbase-0.90.0/bin$

Re: HBase is not ready for Primetime

2011-04-12 Thread Gary Helmling
Robert, You can stop the daemons individually on each node: bin/hbase-daemon.sh stop master bin/hbase-daemon.sh stop regionserver Use this to stop the processes that can be cleanly shutdown. Then let's look at which processes are still hanging and what the logs of the hanging processes are

Re: HBase is not ready for Primetime

2011-04-12 Thread Robert Gonzalez
A bunch of this in the master log: 2011-04-12 12:38:23,771 WARN org.apache.hadoop.hbase.master.CatalogJanitor: REGIONINFO_QUALIFIER is empty in keyvalues={urlhashcopy,E3208173766FDD7C01FE9633E281ED0A,1296085183252.7501ae2b7e933057ea12610c4ec6d001./info:server/1296142856167/Put/vlen=41,

Re: HBase is not ready for Primetime

2011-04-12 Thread Robert Gonzalez
Ok, deleted logs that master was complaining about, restarted master only. Seemed to be stable after a bunch of the messages like the one below, then restarted regionservers, sans the one that gave me trouble this morning. Now seems to be up and running again. I don't trust it, seen this kind

Re: HBase is not ready for Primetime

2011-04-12 Thread Stack
On Tue, Apr 12, 2011 at 10:52 AM, Robert Gonzalez g...@maxpointinteractive.com wrote: Ok, deleted logs that master was complaining about, restarted master only.  Seemed to be stable after a bunch of the messages like the one below, then restarted regionservers, sans the one that gave me trouble