Can you do 'ls /hbase' in zkcli and pastebin the output ? Cheers
On Mon, Apr 27, 2015 at 7:45 AM, Bo Fu <[email protected]> wrote: > Hi Ted, > > I used bin/hbase zkcli to check, and the result was: > Connecting to pc277.emulab.net<http://pc277.emulab.net>:2181, > pc273.emulab.net<http://pc273.emulab.net>:2181,pc312.emulab.net< > http://pc312.emulab.net>:2181 > 2015-04-27 08:42:30,523 INFO [main] zookeeper.ZooKeeper: Client > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT > 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client > environment:host.name=pc322.emulab.net<http://pc322.emulab.net> > 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client > environment:java.version=1.7.0_79 > 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client > environment:java.vendor=Oracle Corporation > 2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client > environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre > 2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client > environment:java.class.path=/proj/ucare/bo/hbase-1.0.0/bin/../conf:/usr/lib/jvm/java-7-openjdk-amd64//lib/tools.jar:/proj/ucare/bo/hbase-1.0.0/bin/..:/proj/ucare/bo/hbase-1.0.0/bin/../lib/activation-1.1.jar:/pro………….. > 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib > 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client > environment:java.io.tmpdir=/tmp > 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client > environment:java.compiler=<NA> > 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client > environment:os.name=Linux > 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client > environment:os.arch=amd64 > 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client > environment:os.version=3.2.0-56-generic > 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client > environment:user.name=BF > 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client > environment:user.home=/users/BF > 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client > environment:user.dir=/proj/ucare/bo/hbase-1.0.0 > 2015-04-27 08:42:30,532 INFO [main] zookeeper.ZooKeeper: Initiating > client connection, connectString=pc277.emulab.net<http://pc277.emulab.net > >:2181,pc273.emulab.net<http://pc273.emulab.net>:2181,pc312.emulab.net< > http://pc312.emulab.net>:2181 sessionTimeout=30000 > watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@920d014 > Welcome to ZooKeeper! > 2015-04-27 08:42:30,694 INFO [main-SendThread(pc277.emulab.net< > http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket > connection to server pc277.emulab.net/155.98.39.77:2181< > http://pc277.emulab.net/155.98.39.77:2181>. Will not attempt to > authenticate using SASL (unknown error) > 2015-04-27 08:42:30,718 INFO [main-SendThread(pc277.emulab.net< > http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection > established to pc277.emulab.net/155.98.39.77:2181< > http://pc277.emulab.net/155.98.39.77:2181>, initiating session > JLine support is enabled > 2015-04-27 08:42:30,760 INFO [main-SendThread(pc277.emulab.net< > http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Session > establishment complete on server pc277.emulab.net/155.98.39.77:2181< > http://pc277.emulab.net/155.98.39.77:2181>, sessionid = > 0x14cfb53f8de0000, negotiated timeout = 30000 > > WATCHER:: > > WatchedEvent state:SyncConnected type:None path:null > [zk: pc277.emulab.net<http://pc277.emulab.net>:2181,pc273.emulab.net< > http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net>:2181(CONNECTED) > 0] > > > It seems that zookeeper is running correctly? > > Bo > > > On Apr 27, 2015, at 9:19 AM, Ted Yu <[email protected]<mailto: > [email protected]>> wrote: > > zkCli.sh is provided by zookeeper. > > See http://hbase.apache.org/book.html#quickstart for standalone HBase > deployment. > > Cheers > > On Mon, Apr 27, 2015 at 6:53 AM, Bo Fu <[email protected]<mailto: > [email protected]>> wrote: > > Hi Ted, > > I can’t find zkCli.sh. I’m using the zookeeper provided by HBase. > > So bad. Should I use my own zookeeper? > > Bo > > > On Apr 26, 2015, at 3:23 PM, Ted Yu <[email protected]<mailto: > [email protected]>> wrote: > > Can you use zkCli.sh to check the contents under /hbase ? > > Please check zookeeper logs to see if quorum was running fine. > > bq. [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>< > http://pc273.emulab.net>:2181)] > > Please remove the '<http://...>' from hbase.zookeeper.quorum config. > > Cheers > > On Sun, Apr 26, 2015 at 11:08 AM, Bo Fu <[email protected]<mailto: > [email protected]>> wrote: > > Hi all, > > I have problem starting a cluster of 1 master and 3 region server. When > I > started the cluster, the HMaster and HRegionserver will automatically > exit. > > My abase-site.xml: > <configuration> > <property> > <name>hbase.master</name> > <value>hadoopmaster:60000</value> > </property> > <property> > <name>hbase.rootdir</name> > <value>hdfs://hadoopmaster:9000/hbase</value> > </property> > <property> > <name>hbase.zookeeper.property.dataDir</name> > <value>/proj/ucare/bo/hadoop_data/zookeeper</value> > </property> > <property> > <name>hbase.zookeeper.quorum</name> > <value>pc225.emulab.net<http://pc225.emulab.net><http://pc225.emulab.net > >,pc273.emulab.net<http://pc273.emulab.net>< > http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net>< > http://pc210.emulab.net > </value> > <description>The directory shared by RegionServers. > </description> > </property> > <property> > <name>hbase.zookeeper.property.clientPort</name> > <value>2181</value> > </property> > <property> > <name>hbase.cluster.distributed</name> > <value>true</value> > </property> > <property> > <name>dfs.replication</name> > <value>1</value> > </property> > </configuration> > > > Log file: > > 2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net< > http://pc273.emulab.net>< > http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket > connection to server pc273.emulab.net/155.98.39.73:2181< > http://pc273.emulab.net/155.98.39.73:2181>< > http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to > authenticate using SASL (unknown error) > 2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net< > http://pc273.emulab.net>< > http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection > established to pc273.emulab.net/155.98.39.73:2181< > http://pc273.emulab.net/155.98.39.73:2181>< > http://pc273.emulab.net/155.98.39.73:2181>, initiating session > 2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net< > http://pc273.emulab.net>< > http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed > socket, > closing socket connection and attempting reconnect > 2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net< > http://pc225.emulab.net>< > http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket > connection to server pc225.emulab.net/155.98.39.25:2181< > http://pc225.emulab.net/155.98.39.25:2181>< > http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to > authenticate using SASL (unknown error) > 2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net< > http://pc225.emulab.net>< > http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection > established to pc225.emulab.net/155.98.39.25:2181< > http://pc225.emulab.net/155.98.39.25:2181>< > http://pc225.emulab.net/155.98.39.25:2181>, initiating session > 2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net< > http://pc225.emulab.net>< > http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed > socket, > closing socket connection and attempting reconnect > 2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net< > http://pc332.emulab.net>< > http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket > connection to server pc332.emulab.net/155.98.39.132:2181< > http://pc332.emulab.net/155.98.39.132:2181>< > http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to > authenticate using SASL (unknown error) > 2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net< > http://pc332.emulab.net>< > http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session > timed out, have not heard from server in 30025ms for sessionid 0x0, > closing > socket connection and attempting reconnect > 2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper: > Possibly transient ZooKeeper, quorum=pc273.emulab.net< > http://pc273.emulab.net>< > http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net>< > http://pc225.emulab.net > :2181,pc332.emulab.net<http://pc332.emulab.net><http://pc332.emulab.net > >:2181, > exception=org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase > 2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper: > ZooKeeper create failed after 4 attempts > 2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster > at > > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982) > at > > > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198) > at > > > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) > at > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512) > at > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491) > at > > > org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252) > at > > > org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230) > at > > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174) > at > > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167) > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531) > at > org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > > Can someone helps me figure out the problem? Is that because the number > of > region server and zookeeper nodes are incorrect? > > Thanks a lot! > > Bo > > > > > > >
