Unusual exception
I started seeing a bunch of these exceptions. What do these mean? 2010-10-13 14:01:33,426 - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:5001:nioserverc...@606] - EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket 2010-10-13 14:01:33,426 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:5001:nioserverc...@1286] - Closed socket connection for client /10.138.34.195:55738 (no session established for client) 2010-10-13 14:01:33,426 - DEBUG [CommitProcessor:1:finalrequestproces...@78] - Processing request:: sessionid:0x12b9d1f8b907a44 type:closeSession cxid:0x0 zxid:0x600193996 txntype:-11 reqpath:n/a 2010-10-13 14:01:33,427 - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:5001:nioserverc...@606] - EndOfStreamException: Unable to read additional data from client sessionid 0x12b9d1f8b907a5d, likely client has closed socket 2010-10-13 14:01:33,427 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:5001:nioserverc...@1286] - Closed socket connection for client /10.138.34.195:55979 which had sessionid 0x12b9d1f8b907a5d 2010-10-13 14:01:33,427 - DEBUG [QuorumPeer:/0.0.0.0:5001 :commitproces...@159] - Committing request:: sessionid:0x52b90ab45bd51af type:createSession cxid:0x0 zxid:0x600193cf9 txntype:-10 reqpath:n/a 2010-10-13 14:01:33,427 - DEBUG [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:5001:nioserverc...@1302] - ignoring exception during output shutdown java.net.SocketException: Transport endpoint is not connected at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) at org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1298) at org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609) at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262) 2010-10-13 14:01:33,428 - DEBUG [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:5001:nioserverc...@1310] - ignoring exception during input shutdown java.net.SocketException: Transport endpoint is not connected at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) at org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306) at org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609) at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262) 2010-10-13 14:01:33,428 - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:5001:nioserverc...@606] - EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket 2010-10-13 14:01:33,428 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:5001:nioserverc...@1286] - Closed socket connection for client /10.138.34.195:55731 (no session established for client)
Membership using ZK
This is what I have going: I have a bunch of 200 nodes come up and create an ephemeral entry under a znode names /Membership. When nodes are detected dead the node associated with the dead node under /Membership is deleted and watch delivered to the rest of the members. Now there are circumstances a node A is deemed dead while the process is still up and running on A. It is a false detection which I need to probably deal with. How do I deal with this situation? Over time false detections delete all the entries underneath the /Membership znode even though all processes are up and running. So my questions are: Would the watches be pushed out to the node that is falsely deemed dead? If so I can have that process recreate the ephemeral znode underneath /Membership. If a node leaves a watch and then truly crashes. When it comes back up would it get watches it missed during the interim period? In any case how do watches behave in the event of false/true failure detection? Thanks A
Re: Membership using ZK
Would my watcher get invoked on this ConnectionLoss event? If so I am thinking I will check for KeeperState.Disconnected and reset my state. Is my understanding correct? Please advice. Thanks Avinash On Tue, Oct 12, 2010 at 10:45 AM, Benjamin Reed br...@yahoo-inc.com wrote: ZooKeeper considers a client dead when it hasn't heard from that client during the timeout period. clients make sure to communicate with ZooKeeper at least once in 1/3 the timeout period. if the client doesn't hear from ZooKeeper in 2/3 the timeout period, the client will issue a ConnectionLoss event and cause outstanding requests to fail with a ConnectionLoss. So, if ZooKeeper decides a process is dead, the process will get a ConnectionLoss event. Once ZooKeeper decides that a client is dead, if the client reconnects, the client will get a SessionExpired. Once a session is expired, the expired handle will become useless, so no new requests, no watches, etc. The bottom line is if your process gets a process expired, you need to treat that process as expired and recover by creating a new zookeeper handle (possibly by restarting the process) and resetup your state. ben On 10/12/2010 09:54 AM, Avinash Lakshman wrote: This is what I have going: I have a bunch of 200 nodes come up and create an ephemeral entry under a znode names /Membership. When nodes are detected dead the node associated with the dead node under /Membership is deleted and watch delivered to the rest of the members. Now there are circumstances a node A is deemed dead while the process is still up and running on A. It is a false detection which I need to probably deal with. How do I deal with this situation? Over time false detections delete all the entries underneath the /Membership znode even though all processes are up and running. So my questions are: Would the watches be pushed out to the node that is falsely deemed dead? If so I can have that process recreate the ephemeral znode underneath /Membership. If a node leaves a watch and then truly crashes. When it comes back up would it get watches it missed during the interim period? In any case how do watches behave in the event of false/true failure detection? Thanks A
Re: What does this mean?
Thanks Ben. I am not mixing processes of different clusters. I just double checked that. I have ZK deployed in a 5 node cluster and I have 20 observers. I just started the 5 node cluster w/o starting the observers. I still the same issue. Now my cluster won't start up. So what is the correct workaround to get this going? How can I find out who the leader is and who the follower to get more insight? Thanks A On Sun, Oct 10, 2010 at 8:33 AM, Benjamin Reed br...@yahoo-inc.com wrote: this usually happens when a follower closes its connection to the leader. it is usually caused by the follower shutting down or failing. you may get further insight by looking at the follower logs. you should really run with timestamps on so that you can correlate the logs of the leader and follower. on thing that is strange is the wide divergence between zxid of follower and leader. are you mixing processes of different clusters? ben From: Avinash Lakshman [avinash.laksh...@gmail.com] Sent: Sunday, October 10, 2010 8:18 AM To: zookeeper-user Subject: What does this mean? I see this exception and the servers not doing anything. java.io.IOException: Channel eof at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) ERROR - 124554051584(higestZxid) 21477836646(next log) for type -11 WARN - Sending snapshot last zxid of peer is 0xe zxid of leader is 0x1e WARN - Sending snapshot last zxid of peer is 0x18 zxid of leader is 0x1eg WARN - Sending snapshot last zxid of peer is 0x5002dc766 zxid of leader is 0x1e WARN - Sending snapshot last zxid of peer is 0x1c zxid of leader is 0x1e ERROR - Unexpected exception causing shutdown while sock still open java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:78) at java.io.DataOutputStream.writeInt(DataOutputStream.java:180) at org.apache.jute.BinaryOutputArchive.writeInt(BinaryOutputArchive.java:55) at org.apache.zookeeper.data.StatPersisted.serialize(StatPersisted.java:116) at org.apache.zookeeper.server.DataNode.serialize(DataNode.java:167) at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123) at org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:967) at org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982) at org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982) at org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982) at org.apache.zookeeper.server.DataTree.serialize(DataTree.java:1031) at org.apache.zookeeper.server.util.SerializeUtils.serializeSnapshot(SerializeUtils.java:104) at org.apache.zookeeper.server.ZKDatabase.serializeSnapshot(ZKDatabase.java:426) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:331) WARN - *** GOODBYE /10.138.34.212:33272 Avinash
snapshots
Hi All Are snapshots serialized dumps of the DataTree taken whenever a log rolls over? So when a server goes down and comes back up does it construct the data tree from the snapshots? What if I am running this on a machine with SSD as extended RAM how does it affect anything? Cheers A
Changing configuration
Suppose I have a 3 node ZK cluster composed of machines A, B and C. Now for whatever reason I lose C forever and the machine needs to be replaced. How do I handle this situation? Update the config with D in place of C and restart the cluster? Also if I am interested in read just the ZAB portions which packages should I be looking at? Cheers A
Too many connections
I find this happening in my observers node in the logs. The observers are running in a different data center from where the ZK non-observers are running. The only way to fix this seems to be restarting. How can I start addressing this? Here is the stack trace. Too many connections from /10.30.84.207 - max is 10 WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:200) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089) Please advice. Cheers Avinash
Re: Too many connections
Thanks Patrick. But what does this mean? I see the log on server A telling me Too many connections from A - default is 10. Too many connection from A to whom? I do not see who the other end of the connection is. Cheers Avinash On Tue, Oct 5, 2010 at 9:27 AM, Patrick Hunt ph...@apache.org wrote: See this configuration param in the docs maxClientCnxns: http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration Patrick On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: I find this happening in my observers node in the logs. The observers are running in a different data center from where the ZK non-observers are running. The only way to fix this seems to be restarting. How can I start addressing this? Here is the stack trace. Too many connections from /10.30.84.207 - max is 10 WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:200) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089) Please advice. Cheers Avinash
Re: Too many connections
So shouldn't all servers in another DC just have one session? So even if I have 50 observers in another DC that should be 50 sessions established since the IP doesn't change correct? Am I missing something? In some ZK clients I see the following exception even though they are in the same DC. WARN - Session 0x0 for server msgzkapp013.abc.com/10.138.43.219:5001, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:200) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089) WARN - Session 0x0 for server msgzkapp012.abc.com/10.138.42.219:5001, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:851) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089) What might be happening in this case? Cheers Avinash On Tue, Oct 5, 2010 at 9:47 AM, Patrick Hunt ph...@apache.org wrote: A (/10.30.84.207 a zookeeper client) is attempting to establish more then 10 sessions to the ZooKeeper server where you got the log. This can be caused by a bug in user code (we've seen bugs where incorrectly implemented ZK clients attempt to create an infinite number of sessions, which essentially DOS the service, so we added the maxClientCnxn default limit of 10). Often users see this problem when they are trying to simulate a real environment - they run a simulated set of clients sessions (10) from a single host (ip) hitting the servers. However in your case I'm guessing that it has something to do with this The observers are running in a different data center from where the ZK non-observers are running. Could you have a NAT or some other networking configuration that makes all the observers seem to be coming from the same IP address? Patrick On Tue, Oct 5, 2010 at 9:33 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Thanks Patrick. But what does this mean? I see the log on server A telling me Too many connections from A - default is 10. Too many connection from A to whom? I do not see who the other end of the connection is. Cheers Avinash On Tue, Oct 5, 2010 at 9:27 AM, Patrick Hunt ph...@apache.org wrote: See this configuration param in the docs maxClientCnxns: http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration Patrick On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: I find this happening in my observers node in the logs. The observers are running in a different data center from where the ZK non-observers are running. The only way to fix this seems to be restarting. How can I start addressing this? Here is the stack trace. Too many connections from /10.30.84.207 - max is 10 WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:200) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089) Please advice. Cheers Avinash
Re: Zookeeper on 60+Gb mem
I have run it over 5 GB of heap with over 10M znodes. We will definitely run it with over 64 GB of heap. Technically I do not see any limitiation. However I will the experts chime in. Avinash On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar maha...@yahoo-inc.comwrote: Hi Maarteen, I definitely know of a group which uses around 3GB of memory heap for zookeeper but never heard of someone with such huge requirements. I would say it definitely would be a learning experience with such high memory which I definitely think would be very very useful for others in the community as well. Thanks mahadev On 10/5/10 11:03 AM, Maarten Koopmans maar...@vrijheid.net wrote: Hi, I just wondered: has anybody ever ran zookeeper to the max on a 68GB quadruple extra large high memory EC2 instance? With, say, 60GB allocated or so? Because EC2 with EBS is a nice way to grow your zookeeper cluster (data on the ebs columes, upgrade as your memory utilization grows) - I just wonder what the limits are there, or if I am foing where angels fear to tread... --Maarten
ZooDefs.Sync
What is a Sync OpCode stand for? In particular what does a LearnerSyncRequest represent? Thanks Avinash
Zookeeper CLI
I have a 5 server ZK cluster. I want to connect to it using the CLI from some remote machine. Is there any particular set up that I need to connect? I am running it as zkCli.sh -server server name where the server name is one of the servers in the ZK cluster. Is this correct? I can get a string of Connect Exceptions as shown below: 2010-09-07 16:46:07,955 - WARN [main-SendThread(msgzkapp001.ash2.facebook.com:2181 ):clientcnxn$sendthr...@1120] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:672) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-07 16:46:09,141 - INFO [main-SendThread(msgzkapp001.ash2.facebook.com:2181 ):clientcnxn$sendthr...@1000] - Opening socket connection to server msgzkapp001.ash2.facebook.com/10.138.31.220:2181 2010-09-07 16:46:09,220 - WARN [main-SendThread(msgzkapp001.ash2.facebook.com:2181 ):clientcnxn$sendthr...@1120] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:672) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) Cheers Avinash
Re: Zookeeper CLI
Never mind. It was my bad. I figured it out. Thanks Avinash On Tue, Sep 7, 2010 at 4:56 PM, Henry Robinson he...@cloudera.com wrote: Are you running ZK on the standard ports (in particular, client port 2181)? Can you telnet into those servers on that port and issue an ruok command? Henry On 7 September 2010 16:48, Avinash Lakshman avinash.laksh...@gmail.com wrote: I have a 5 server ZK cluster. I want to connect to it using the CLI from some remote machine. Is there any particular set up that I need to connect? I am running it as zkCli.sh -server server name where the server name is one of the servers in the ZK cluster. Is this correct? I can get a string of Connect Exceptions as shown below: 2010-09-07 16:46:07,955 - WARN [main-SendThread(msgzkapp001.ash2.facebook.com:2181 ):clientcnxn$sendthr...@1120] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:672) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-07 16:46:09,141 - INFO [main-SendThread(msgzkapp001.ash2.facebook.com:2181 ):clientcnxn$sendthr...@1000] - Opening socket connection to server msgzkapp001.ash2.facebook.com/10.138.31.220:2181 2010-09-07 16:46:09,220 - WARN [main-SendThread(msgzkapp001.ash2.facebook.com:2181 ):clientcnxn$sendthr...@1120] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:672) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) Cheers Avinash -- Henry Robinson Software Engineer Cloudera 415-994-6679
Logs and in memory operations
Hi All From my understanding when a znode is updated/created a write happens into the local transaction logs and then some in-memory data structure is updated to serve the future reads. Where in the source code can I find this? Also how can I decide when it is ok for me to delete the logs off disk? Please advice. Cheers Avinash
Adding observers
Hi All I have two questions regarding Observers in ZK. (1) Is it possible to increase the number of observers in the cluster dynamically? (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? Any insight would be very helpful. Thanks A
Re: Adding observers
Sorry as an addendum I had one more question. (1) If I snapshot the data on other observer machines will I be able to bootstrap new observers with it? Given that writes are like a one time thing. Cheers A On Wed, Jul 21, 2010 at 10:30 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Hi All I have two questions regarding Observers in ZK. (1) Is it possible to increase the number of observers in the cluster dynamically? (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? Any insight would be very helpful. Thanks A
Re: Adding observers
Any example scripts for the rolling restart technique that anyone would be kind enough to share? Thanks Avinash On Wed, Jul 21, 2010 at 10:44 AM, Henry Robinson he...@cloudera.com wrote: Hi Avinash - (1) Is it possible to increase the number of observers in the cluster dynamically? Not really - you can use the same rolling-restart technique that is often mentioned on this list for adding servers to the ensemble, but you can't add them and expect that ZK will auto-find them. (2) How many observers can I add given that I will seldom write into the cluster but will have a lot of reads coming into the system? Can I run a cluster with say 100 observers? This will be possible, but there is some overhead in communicating with observers as well as normal voting followers. However, given that writes are rare, perhaps this kind of overhead would be acceptable? Henry Any insight would be very helpful. Thanks A -- Henry Robinson Software Engineer Cloudera 415-994-6679
What does this exception mean?
Hi All I run into this periodically. I am curious to know what this means, why would this happen and how am I to react to it programmatically. org.apache.thrift.TException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /Config/Stats/count at com.abc.service.MyService.handleAll(MyService.java:223) at com.abc.service.MyService.assign(AtlasService.java:344) at com.abc.service.MyService.assign(AtlasService.java:364) at com.abc.service.MyService.assignAll(AtlasService.java:385) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /Config/Stats/count at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:518) Please advice. Thanks Avinash
Recovery protocol
Hi All Let's say I have a Zookeeper cluster with nodes A, B, C, D and E. Let's assume A is the leader. Now let us assume after a few writes have taken place the node B crashes. When it comes back up what is the recovery protocol? Does it join the cluster immediately and start taking writes and reads while it is catching up on writes that it lost when it was down? Cheers Avinash
Usage of myId
Why is this important? What breaks down if I have 2 servers with the same myId? Cheers A
Re: API for node entry to the cluster.
I would prefer not restarting. Start/Stop the new/old process and then start a round of consensus for adding/removing a machine. I guess if one can do that then there is stopping of process required. Am I missing something here? A On Thu, Nov 5, 2009 at 11:14 AM, Ted Dunning ted.dunn...@gmail.com wrote: It is pretty easy to do by hand and is pretty easy to script (it was for us). The process for adding is a) configure and start a new node b) re-configure and restart each existing node in turn to know about the new node you now have a larger cluster. To drop a node, a) reconfigure and restart each surviving node b) kill the node(s) that is(are) leaving you now have a smaller cluster. On Thu, Nov 5, 2009 at 11:02 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Hi All Is it possible to remove nodes and add nodes dynamically to the ZK cluster via API? Any plans in the future to do this? TIA A -- Ted Dunning, CTO DeepDyve
Re: API for node entry to the cluster.
Ah. Sorry. But I got it. I guess what Henry is working on is what I am looking for. Thanks A On Thu, Nov 5, 2009 at 11:32 AM, Ted Dunning ted.dunn...@gmail.com wrote: You have two options. What Henry is talking about is a sophisticated update to ZK internals that does a very nice job of handling all the possible split-brain issues and other pathologies that can result from growing or shrinking a cluster. His patch isn't ready yet, but when it is, it should be really nice. What I am talking about is pretty tame hack that allows you to get 99% of the result needed with very little effort. It can be done today. Restarting servers one at a time is conventionally used for minor releases of ZK and can be (ab)used to allow configuration changes. I don't quite follow what you are saying in your second and third sentences so I can't answer the question in your fourth. Can you clarify what you mean? I halfway suspect you dropped a word somewhere. On Thu, Nov 5, 2009 at 11:24 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: I would prefer not restarting. Start/Stop the new/old process and then start a round of consensus for adding/removing a machine. I guess if one can do that then there is stopping of process required. Am I missing something here? A On Thu, Nov 5, 2009 at 11:14 AM, Ted Dunning ted.dunn...@gmail.com wrote: It is pretty easy to do by hand and is pretty easy to script (it was for us). The process for adding is a) configure and start a new node b) re-configure and restart each existing node in turn to know about the new node you now have a larger cluster. To drop a node, a) reconfigure and restart each surviving node b) kill the node(s) that is(are) leaving you now have a smaller cluster. On Thu, Nov 5, 2009 at 11:02 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Hi All Is it possible to remove nodes and add nodes dynamically to the ZK cluster via API? Any plans in the future to do this? TIA A -- Ted Dunning, CTO DeepDyve -- Ted Dunning, CTO DeepDyve
API for node entry to the cluster.
Hi All Is it possible to remove nodes and add nodes dynamically to the ZK cluster via API? Any plans in the future to do this? TIA A
Watches
Hi All Is it possible to have 100K machines register for a watch on a znode? I mean theoritically yes it should work but ZK scale to these many instances when it comes to delivering watch notifications? Perhaps no one has practical experience in dealing with this but is there any fundamental limitation that I should be aware of? These 100K machines are only interested in receiving notifications. Cheers Avinash
Watches - is delivery guaranteed?
Hi All Suppose I have a znode, say /Me, and have three nodes A, B and C who have set watches on this znode. Now suppose some process changes some value on /Me then watches get delivered to A, B and C. Now if at that instant of time C were down I could always read the znode on coming back up. However if C were partitioned away would the watch be delivered in a guaranteed fashion when the partition heals? Or are watches best effort delivery? If a znode is changed N times am I going to get N watches delivered one for each change or could I be in a situation where I could get only a few watches but the final change will be delivered to me. The reason for this is that I have a situation where I set a watch on znode /Parent underneath which many znodes /Parent/Child-i could be added one per process in my cluster on startup. I need to get all the znodes under /Parent when startup is complete. Am I guaranteed to get the watch which contains all the children provided all the processes do successfully put an entry in ZK. Is this clear? If not I can try to articulate it better. Cheers Avinash
Latest download of Zookeeper binaries
Hi Where can I find the latest binaries of Zookeeper? None of the download links off the Apache site seem to work. Avinash
