OK. That means that the hbase master will start ZK. It should still be accessible on port 2181 if hb master is running.
On Thu, Jan 20, 2011 at 2:12 PM, mike anderson <[email protected]>wrote: > I'm not that familiar with ZK, but I have it set to: > > # Tell HBase whether it should manage it's own instance of Zookeeper or > not. > export HBASE_MANAGES_ZK=true > > and > > <property> > <name>hbase.cluster.distributed</name> > <value>true</value> > <description>The mode the cluster will be in. Possible values are > false: standalone and pseudo-distributed setups with managed Zookeeper > true: fully-distributed with unmanaged Zookeeper Quorum (see > hbase-env.sh) > </description> > </property> > <property> > > > I'll look at the diagnostic tools. Thanks, > > Mike > > > On Thu, Jan 20, 2011 at 5:09 PM, Ted Dunning <[email protected]> > wrote: > > > Are you running a standalone ZK cluster? > > > > If so, are you doing the standard ZK diagnostic things? > > > > ZK supports several simple checks which you can access by sending special > > four byte sequences to the standard client port. > > > > See > > > > > http://hadoop.apache.org/zookeeper/docs/r3.2.0/zookeeperAdmin.html#sc_zkCommands > > > > < > > > http://hadoop.apache.org/zookeeper/docs/r3.2.0/zookeeperAdmin.html#sc_zkCommands > > >(but > > watch out with the echo idiom ... some versions of ZK or nc may close the > > connection too quickly to get teh reply) > > > > > > > > On Thu, Jan 20, 2011 at 1:52 PM, mike anderson <[email protected] > > >wrote: > > > > > Alright I'm stumped here. It looks like Zookeeper isn't playing nice > and > > > causing the master to not start. Any suggestions? > > > > > > > > > I'm seeing a lot of these: 2011-01-20 16:47:55,744 INFO > > > org.apache.zookeeper.ClientCnxn: Unable to read additional data from > > server > > > sessionid 0x0, likely server has closed socket, closing socket > connection > > > and attempting reconnect > > > > > > Which looks bad, full log below. > > > > > > -mike > > > > > > > > > > > > > > > Thu Jan 20 16:47:47 EST 2011 Starting master on carr.host.com > > > ulimit -n 20480 > > > 2011-01-20 16:47:47,571 INFO > org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: > > > Initializing RPC Metrics with hostName=HMaster, port=60000 > > > 2011-01-20 16:47:47,628 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > Responder: starting > > > 2011-01-20 16:47:47,630 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > listener on 60000: starting > > > 2011-01-20 16:47:47,630 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 0 on 60000: starting > > > 2011-01-20 16:47:47,631 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 1 on 60000: starting > > > 2011-01-20 16:47:47,631 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 2 on 60000: starting > > > 2011-01-20 16:47:47,631 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 3 on 60000: starting > > > 2011-01-20 16:47:47,631 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 4 on 60000: starting > > > 2011-01-20 16:47:47,631 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 5 on 60000: starting > > > 2011-01-20 16:47:47,631 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 6 on 60000: starting > > > 2011-01-20 16:47:47,631 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 7 on 60000: starting > > > 2011-01-20 16:47:47,632 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 8 on 60000: starting > > > 2011-01-20 16:47:47,633 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > Server > > > handler 9 on 60000: starting > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 > > GMT > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:host.name=carr.host.com > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:java.version=1.6.0_07 > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:java.vendor=Sun Microsystems Inc. > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:java.home=/usr/java/jre1.6.0_07 > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > > > > > > > environment:java.class.path=/usr/local/hbase/bin/../conf:/usr/java/jre1.6.0_07/lib/tools.jar:/usr/local/hbase/bin/..:/usr/local/hbase/bin/../hbase-0.90.0.jar:/usr/local/hbase/bin/../hbase-0.90.0-tests.jar:/usr/local/hbase/bin/../lib/activation-1.1.jar:/usr/local/hbase/bin/../lib/ant-1.6.5.jar:/usr/local/hbase/bin/../lib/ant-1.7.1.jar:/usr/local/hbase/bin/../lib/ant-launcher-1.7.1.jar:/usr/local/hbase/bin/../lib/asm-3.1.jar:/usr/local/hbase/bin/../lib/avro-1.3.3.jar:/usr/local/hbase/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase/bin/../lib/commons-codec-1.4.jar:/usr/local/hbase/bin/../lib/commons-el-1.0.jar:/usr/local/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase/bin/../lib/commons-lang-2.5.jar:/usr/local/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/local/hbase/bin/../lib/core-3.1.1.jar:/usr/local/hbase/bin/../lib/guava-r06.jar:/usr/local/hbase/bin/../lib/hadoop-0.20.2-core.jar:/usr/local/hbase/bin/../lib/hbase-0.90.0.jar:/usr/local/hbase/bin/../lib/hsqldb-1.8.0.10.jar:/usr/local/hbase/bin/../lib/jackson-core-asl-1.5.5.jar:/usr/local/hbase/bin/../lib/jackson-jaxrs-1.5.5.jar:/usr/local/hbase/bin/../lib/jackson-mapper-asl-1.4.2.jar:/usr/local/hbase/bin/../lib/jackson-xc-1.5.5.jar:/usr/local/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/local/hbase/bin/../lib/jaxb-impl-2.1.12.jar:/usr/local/hbase/bin/../lib/jersey-core-1.4.jar:/usr/local/hbase/bin/../lib/jersey-json-1.4.jar:/usr/local/hbase/bin/../lib/jersey-server-1.4.jar:/usr/local/hbase/bin/../lib/jets3t-0.7.1.jar:/usr/local/hbase/bin/../lib/jettison-1.1.jar:/usr/local/hbase/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase/bin/../lib/jline-0.9.94.jar:/usr/local/hbase/bin/../lib/jruby-complete-1.0.3.jar:/usr/local/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsr311-api-1.1.1.jar:/usr/local/hbase/bin/../lib/junit-3.8.1.jar:/usr/local/hbase/bin/../lib/kfs-0.3.jar:/usr/local/hbase/bin/../lib/log4j-1.2.16.jar:/usr/local/hbase/bin/../lib/oro-2.0.8.jar:/usr/local/hbase/bin/../lib/paranamer-2.2.jar:/usr/local/hbase/bin/../lib/paranamer-ant-2.2.jar:/usr/local/hbase/bin/../lib/paranamer-generator-2.2.jar:/usr/local/hbase/bin/../lib/protobuf-java-2.3.0.jar:/usr/local/hbase/bin/../lib/qdox-1.10.1.jar:/usr/local/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/bin/../lib/slf4j-api-1.5.8.jar:/usr/local/hbase/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/local/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/local/hbase/bin/../lib/thrift-0.2.0.jar:/usr/local/hbase/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase/bin/../lib/zookeeper-3.3.2.jar > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > > > > > > > environment:java.library.path=/usr/java/jre1.6.0_07/lib/amd64/server:/usr/java/jre1.6.0_07/lib/amd64:/usr/java/jre1.6.0_07/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:java.io.tmpdir=/tmp > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:java.compiler=<NA> > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:os.name=Linux > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:os.arch=amd64 > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:os.version=2.6.18-53.1.21.el5 > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:user.name=pubget > > > 2011-01-20 16:47:47,651 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:user.home=/home/pubget > > > 2011-01-20 16:47:47,652 INFO org.apache.zookeeper.ZooKeeper: Client > > > environment:user.dir=/home/pubget/hbase-0.90.0 > > > 2011-01-20 16:47:47,652 INFO org.apache.zookeeper.ZooKeeper: Initiating > > > client connection, connectString=ghetto163.host.com:2181, > > > ghetto162.host.com:2181,ghetto161.host.com:2181, > ghetto160.host.com:2181, > > > ghetto169.host.com:2181,ghetto167.host.com:2181, > ghetto165.host.com:2181, > > > ghetto164.host.com:2181 sessionTimeout=180000 watcher=master:60000 > > > 2011-01-20 16:47:47,673 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto167.host.com/10.0.16.167:2181 > > > 2011-01-20 16:47:47,678 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto167.host.com/10.0.16.167:2181, > > initiating > > > session > > > 2011-01-20 16:47:47,683 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:47,957 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto163.host.com/10.0.16.163:2181 > > > 2011-01-20 16:47:47,957 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto163.host.com/10.0.16.163:2181, > > initiating > > > session > > > 2011-01-20 16:47:47,958 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:48,906 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto169.host.com/10.0.16.169:2181 > > > 2011-01-20 16:47:48,907 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto169.host.com/10.0.16.169:2181, > > initiating > > > session > > > 2011-01-20 16:47:48,907 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:49,784 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto160.host.com/10.0.16.160:2181 > > > 2011-01-20 16:47:49,785 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto160.host.com/10.0.16.160:2181, > > initiating > > > session > > > 2011-01-20 16:47:49,785 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:50,605 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto162.host.com/10.0.16.162:2181 > > > 2011-01-20 16:47:50,606 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto162.host.com/10.0.16.162:2181, > > initiating > > > session > > > 2011-01-20 16:47:50,607 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:50,885 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto161.host.com/10.0.16.161:2181 > > > 2011-01-20 16:47:50,886 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto161.host.com/10.0.16.161:2181, > > initiating > > > session > > > 2011-01-20 16:47:50,887 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:51,150 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto164.host.com/10.0.16.164:2181 > > > 2011-01-20 16:47:51,150 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto164.host.com/10.0.16.164:2181, > > initiating > > > session > > > 2011-01-20 16:47:51,151 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:52,042 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto165.host.com/10.0.16.165:2181 > > > 2011-01-20 16:47:52,042 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto165.host.com/10.0.16.165:2181, > > initiating > > > session > > > 2011-01-20 16:47:52,043 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:52,832 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto167.host.com/10.0.16.167:2181 > > > 2011-01-20 16:47:52,833 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto167.host.com/10.0.16.167:2181, > > initiating > > > session > > > 2011-01-20 16:47:52,833 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:53,749 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto163.host.com/10.0.16.163:2181 > > > 2011-01-20 16:47:53,750 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto163.host.com/10.0.16.163:2181, > > initiating > > > session > > > 2011-01-20 16:47:53,750 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:54,632 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto169.host.com/10.0.16.169:2181 > > > 2011-01-20 16:47:54,632 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto169.host.com/10.0.16.169:2181, > > initiating > > > session > > > 2011-01-20 16:47:54,633 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:55,254 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto160.host.com/10.0.16.160:2181 > > > 2011-01-20 16:47:55,254 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto160.host.com/10.0.16.160:2181, > > initiating > > > session > > > 2011-01-20 16:47:55,255 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:55,743 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto162.host.com/10.0.16.162:2181 > > > 2011-01-20 16:47:55,743 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto162.host.com/10.0.16.162:2181, > > initiating > > > session > > > 2011-01-20 16:47:55,744 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:56,598 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto161.host.com/10.0.16.161:2181 > > > 2011-01-20 16:47:56,598 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto161.host.com/10.0.16.161:2181, > > initiating > > > session > > > 2011-01-20 16:47:56,599 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:57,696 INFO org.apache.zookeeper.ClientCnxn: Opening > > > socket > > > connection to server ghetto164.host.com/10.0.16.164:2181 > > > 2011-01-20 16:47:57,697 INFO org.apache.zookeeper.ClientCnxn: Socket > > > connection established to ghetto164.host.com/10.0.16.164:2181, > > initiating > > > session > > > 2011-01-20 16:47:57,697 INFO org.apache.zookeeper.ClientCnxn: Unable to > > > read > > > additional data from server sessionid 0x0, likely server has closed > > socket, > > > closing socket connection and attempting reconnect > > > 2011-01-20 16:47:57,802 ERROR > > > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start > master > > > java.lang.RuntimeException: Failed construction of Master: class > > > org.apache.hadoop.hbase.master.HMaster > > > at > > > > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1057) > > > at > > > > > > > > > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142) > > > at > > > > > > > > > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102) > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > > at > > > > > > > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) > > > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1071) > > > Caused by: > org.apache.zookeeper.KeeperException$ConnectionLossException: > > > KeeperErrorCode = ConnectionLoss for /hbase > > > at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > > > at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > > > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637) > > > at > > > > > > > > > org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902) > > > at > > > > > > > > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133) > > > at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:217) > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown > Source) > > > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > > > Source) > > > at java.lang.reflect.Constructor.newInstance(Unknown Source) > > > at > > > > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1052) > > > ... 5 more > > > > > >
