Here is the log since I started it... Wed Oct 7 13:27:26 EDT 2009 Starting master on ip-10-244-9-171 ulimit -n 1024 2009-10-07 13:27:26,404 INFO org.apache.hadoop.hbase.master.HMaster: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Sun Microsystems Inc., vmVersion=14.2-b01 2009-10-07 13:27:26,405 INFO org.apache.hadoop.hbase.master.HMaster: vmInputArguments=[-Xmx2000m, -XX:+HeapDumpOnOutOfMemoryError, -Djava.io.tmpdir=/mnt/tmp, -Dhbase.log.dir=/mnt/apps/hadoop/hbase/bin/../logs, -Dhbase.log.file=hbase-root-master-ip-10-244-9-171.log, -Dhbase.home.dir=/mnt/apps/hadoop/hbase/bin/.., -Dhbase.id.str=root, -Dhbase.root.logger=INFO,DRFA, -Djava.library.path=/mnt/apps/hadoop/hbase/bin/../lib/native/Linux-amd64-64] 2009-10-07 13:27:27,525 INFO org.apache.hadoop.hbase.master.HMaster: Root region dir: s3://hbase2.s3.amazonaws.com:80/hbasedata/-ROOT-/70236052 2009-10-07 13:27:27,751 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Initializing RPC Metrics with hostName=HMaster, port=60000 2009-10-07 13:27:27,827 INFO org.apache.hadoop.hbase.master.HMaster: HMaster initialized on 10.244.9.171:60000 2009-10-07 13:27:27,829 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=Master, sessionId=HMaster 2009-10-07 13:27:27,830 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized 2009-10-07 13:27:27,932 INFO org.mortbay.util.Credential: Checking Resource aliases 2009-10-07 13:27:27,936 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4 2009-10-07 13:27:27,936 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs] 2009-10-07 13:27:28,202 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.webapplicationhand...@3209fa8f 2009-10-07 13:27:28,244 INFO org.mortbay.util.Container: Started WebApplicationContext[/static,/static] 2009-10-07 13:27:28,361 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.webapplicationhand...@b0c0f66 2009-10-07 13:27:28,364 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/] 2009-10-07 13:27:28,636 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.webapplicationhand...@3c2d7440 2009-10-07 13:27:28,638 INFO org.mortbay.util.Container: Started WebApplicationContext[/api,rest] 2009-10-07 13:27:28,639 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60010 2009-10-07 13:27:28,639 INFO org.mortbay.util.Container: Started org.mortbay.jetty.ser...@28b301f2 2009-10-07 13:27:28,640 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server Responder: starting 2009-10-07 13:27:28,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000: starting 2009-10-07 13:27:28,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: starting 2009-10-07 13:27:28,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: starting 2009-10-07 13:27:28,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: starting 2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: starting 2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: starting 2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: starting 2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: starting 2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: starting 2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: starting 2009-10-07 13:27:28,642 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads 2009-10-07 13:27:28,643 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: starting 2009-10-07 13:28:09,519 INFO org.apache.hadoop.hbase.master.RegionManager: in safe mode 2009-10-07 13:28:11,542 INFO org.apache.hadoop.hbase.master.RegionManager: in safe mode 2009-10-07 13:28:13,543 INFO org.apache.hadoop.hbase.master.RegionManager: in safe mode 2009-10-07 13:28:15,545 INFO org.apache.hadoop.hbase.master.RegionManager: in safe mode 2009-10-07 13:28:17,548 INFO org.apache.hadoop.hbase.master.RegionManager: in safe mode 2009-10-07 13:28:19,555 INFO org.apache.hadoop.hbase.master.RegionManager: in safe mode 2009-10-07 13:28:27,834 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned 2009-10-07 13:29:27,832 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned 2009-10-07 13:29:37,593 INFO org.apache.hadoop.hbase.master.RegionManager: in safe mode 2009-10-07 13:30:27,834 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned 2009-10-07 13:31:27,836 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned 2009-10-07 13:32:27,838 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned 2009-10-07 13:33:27,840 INFO org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s) scanned
Ananth T Sarathy On Wed, Oct 7, 2009 at 1:20 PM, stack <[email protected]> wrote: > Thats interesting to hear. Keep us posted. > > HBase asks the filesystem if its in safe mode and if it is, it parks > itself. Here is code from master: > > if (this.fs instanceof DistributedFileSystem) { > // Make sure dfs is not in safe mode > String message = "Waiting for dfs to exit safe mode..."; > while (((DistributedFileSystem) fs).setSafeMode( > FSConstants.SafeModeAction.SAFEMODE_GET)) { > LOG.info(message); > try { > Thread.sleep(this.threadWakeFrequency); > } catch (InterruptedException e) { > //continue > } > } > } > > > Then there is hbase's notion of safemode. It will be in safe mode until it > does initial scan of catalog tables. The master keeps a flag in zookeeper > while its in safemode so regionservers are aware of the state: > > public boolean inSafeMode() { > if (safeMode) { > if(isInitialMetaScanComplete() && regionsInTransition.size() == 0 && > tellZooKeeperOutOfSafeMode()) { > master.connection.unsetRootRegionLocation(); > safeMode = false; > LOG.info("exiting safe mode"); > } else { > LOG.info("in safe mode"); > } > } > return safeMode; > } > > Have you seen the .META. and -ROOT- deploy to regionservers? Have you seen > that these regions being scanned in the master log? (Enable DEBUG if not > already enabled). > > Yours, > ST.Ack > > > On Wed, Oct 7, 2009 at 10:06 AM, Ananth T. Sarathy < > [email protected]> wrote: > > > We have been running Hbase on a s3 filesystem. It's the hbase > regionserver, > > not HDFS since we are using s3. We haven't felt like it's been too slow, > > though the amount of data we are pushing isn't sufficiently large enough > to > > notice yet. > > Ananth T Sarathy > > > > > > On Wed, Oct 7, 2009 at 12:47 PM, stack <[email protected]> wrote: > > > > > HBase or HDFS is in safe mode. My guess is that its the latter. Can > > you > > > figure from HDFS logs why it won't leave safe mode? Usually > > > under-replication or a loss of a large swath of the cluster will flip > on > > > the > > > safe-mode switch. > > > > > > Are you trying to run HBASE on an S3 filesystem? An HBasista tried it > in > > > the past and, FYI, found it insufferably slow. Let us know how it goes > > for > > > you. > > > > > > Thanks, > > > St.Ack > > > > > > On Wed, Oct 7, 2009 at 9:33 AM, Ananth T. Sarathy < > > > [email protected]> wrote: > > > > > > > my regionserver has been stuck in safemode. What can i do to get it > > out > > > > safemode? > > > > > > > > Ananth T Sarathy > > > > > > > > > >
