Setting up a development cluster.
Using Cloudera's latest release which has HBase-20.3.
We have 3 nodes running ZooKeeper which is managed by HBase.
We have a quorum set up.
One of the developers ran in to this error:
10/02/26 07:31:48 WARN zookeeper.ClientCnxn: Exception closing
session 0x0 to sun.nio.ch.selectionkeyi...@6708f8e0java.net.connectexception:
Connection refused at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)10/02/26
07:31:48 WARN zookeeper.ClientCnxn: Ignoring exception
during shutdown inputjava.nio.channels.ClosedChannelException at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) at
sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)10/02/26
07:31:48 WARN zookeeper.ClientCnxn: Ignoring exception
during shutdown outputjava.nio.channels.ClosedChannelException at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at
sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)10/02/26
07:31:49 WARN zookeeper.ZooKeeperWrapper: Failed to create
/hbase -- check quorum servers,
currently=localhost:2181org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
[SNIP]
The strange thing is that we have a quorum but the error says
currently=localhost.
Checking the logs, I found that I had an error ... KeeperErrorCode =
NodeExistsI set the logging to DEBUG and restarted both HBase and Hadoop. (When
both down checked that no zombie processes were left running.)
Now I get the following block in the zookeeper log:2010-02-26 14:35:36,435 WARN
org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing
sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0xfffffffffffffffe
txntype:unknown n/a
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists
at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245)
at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114)
2010-02-26 14:35:36,435 DEBUG
org.apache.zookeeper.server.quorum.CommitProcessor: Processing request::
sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 txntype:-1 n/a
2010-02-26 14:35:36,436 DEBUG org.apache.zookeeper.server.quorum.Leader:
Proposing:: sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019
txntype:-1 n/a
2010-02-26 14:35:36,439 DEBUG org.apache.zookeeper.server.quorum.Leader: Count
for zxid: 0x6300000019 is 1
2010-02-26 14:35:36,445 DEBUG org.apache.zookeeper.server.quorum.Leader: Count
for zxid: 0x6300000019 is 2
2010-02-26 14:35:36,445 DEBUG
org.apache.zookeeper.server.quorum.CommitProcessor: Committing request::
sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 txntype:-1 n/a
2010-02-26 14:35:36,445 DEBUG org.apache.zookeeper.server.quorum.Leader:
outstanding is 0
2010-02-26 14:35:36,445 DEBUG
org.apache.zookeeper.server.FinalRequestProcessor: Processing request::
sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 txntype:-1 n/a
Now this happens when I run a command in the hbase shell. (Like list, status,
...)
I'd like to understand what is happening and what I can do to fix this?
I'm not sure why the user had Zookeeper errors showing local host but the
larger concern is what is happening to cause this?
Earlier I had i/o errors mentioning /hbase/root but they went away when I took
everything down.Since this is still a development machine, I may just blow away
/hbase in HDFS and see if I can come up clean.
TIA.
-Mike
_________________________________________________________________
Hotmail: Free, trusted and rich email service.
http://clk.atdmt.com/GBL/go/201469228/direct/01/
