Re: rolling upgrade 3.2.1 -> 3.3.0
Fantastic. Thanks. On Apr 14, 2010, at 4:26 PM, Mahadev Konar wrote: > Hi Charity, > Looks like you are hitting a bug recently found in 3.3.0. > > https://issues.apache.org/jira/browse/ZOOKEEPER-737 > > > Is the bug, wherein the server does not show the right status. Looks like in > your case the server is running fine but bin/zkserver.sh status is not > returning the right result. > > You can try telnet localhost port and then type stat to get the status on > the server. This bug will be fixed in the bug fix release 3.3.1 which most > probalbly will be released by next week or so. > > Thanks > mahadev > > > > On 4/14/10 3:59 PM, "Charity Majors" wrote: > >> Hi. I'm trying to upgrade a zookeeper cluster from 3.2.1 to 3.3.0, and >> having >> problems. I can't get a 3.3.0 node to successfully join the cluster and stay >> joined. >> >> If I run zkServer.sh status immediately after starting up the newly upgraded >> node, it says the service is probably not running, and shows me this: >> >> >> [char...@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status >> JMX enabled by default >> Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg >> 2010-04-14 22:47:35,574 - INFO >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioservercnxn$fact...@251] - >> Accepted socket connection from /127.0.0.1:40287 >> 2010-04-14 22:47:35,576 - INFO >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@968] - Processing >> stat command from /127.0.0.1:40287 >> 2010-04-14 22:47:35,577 - WARN >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@606] - >> EndOfStreamException: Unable to read additional data from client sessionid >> 0x0, likely client has closed socket >> 2010-04-14 22:47:35,578 - INFO >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@1286] - Closed >> socket >> connection for client /127.0.0.1:40287 (no session established for client) >> Error contacting service. It is probably not running. >> [char...@test-zookeeper001 zookeeper-current]$ 2010-04-14 22:47:35,580 - >> DEBUG >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@1310] - ignoring >> exception during input shutdown >> java.net.SocketException: Transport endpoint is not connected >>at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) >>at >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) >>at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >>at >> org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306) >>at >> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263) >>at >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609) >>at >> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262) >> >> >> If I connect with zkCli.sh, I can list the contents of zookeeper. If I make >> changes to the schema on either of the other two nodes, test-zookeeper002 and >> test-zookeeper003, both of which are running 3.2.1, the changes are reflected >> on test-zookeeper001, which is running 3.3.0. >> >> When I exit zkCli.sh, however, zkServer.sh status starts flapping between >> "Error contacting service. It is probably not running." and "Mode: follower", >> as you can see below. >> >> Any ideas? I'd really rather not have to take the production zookeeper >> cluster down to upgrade if it's not necessary. >> >> Thanks, >> Charity. >> >> >> >> [char...@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status >> JMX enabled by default >> Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg >> 2010-04-14 22:53:16,848 - INFO >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioservercnxn$fact...@251] - >> Accepted socket connection from /127.0.0.1:55284 >> 2010-04-14 22:53:16,849 - INFO >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@968] - Processing >> stat command from /127.0.0.1:55284 >> 2010-04-14 22:53:16,849 - WARN >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@606] - >> EndOfStreamException: Unable to read additional data from client sessionid >> 0x0, likely client has closed socket >> 2010-04-14 22:53:16,850 - INFO >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@1286] - Closed >> socket >> connection for client /127.0.0.1:55284 (no session established for client) >> Error contacting service. It is probably not running. >> 2010-04-14 22:53:16,850 - DEBUG >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@1310] - ignoring >> exception during input shutdown >> java.net.SocketException: Transport endpoint is not connected >>at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) >>at >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) >>at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >>at >> org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306)
Re: rolling upgrade 3.2.1 -> 3.3.0
Hi Charity, Looks like you are hitting a bug recently found in 3.3.0. https://issues.apache.org/jira/browse/ZOOKEEPER-737 Is the bug, wherein the server does not show the right status. Looks like in your case the server is running fine but bin/zkserver.sh status is not returning the right result. You can try telnet localhost port and then type stat to get the status on the server. This bug will be fixed in the bug fix release 3.3.1 which most probalbly will be released by next week or so. Thanks mahadev On 4/14/10 3:59 PM, "Charity Majors" wrote: > Hi. I'm trying to upgrade a zookeeper cluster from 3.2.1 to 3.3.0, and having > problems. I can't get a 3.3.0 node to successfully join the cluster and stay > joined. > > If I run zkServer.sh status immediately after starting up the newly upgraded > node, it says the service is probably not running, and shows me this: > > > [char...@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status > JMX enabled by default > Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg > 2010-04-14 22:47:35,574 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioservercnxn$fact...@251] - > Accepted socket connection from /127.0.0.1:40287 > 2010-04-14 22:47:35,576 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@968] - Processing > stat command from /127.0.0.1:40287 > 2010-04-14 22:47:35,577 - WARN > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@606] - > EndOfStreamException: Unable to read additional data from client sessionid > 0x0, likely client has closed socket > 2010-04-14 22:47:35,578 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@1286] - Closed socket > connection for client /127.0.0.1:40287 (no session established for client) > Error contacting service. It is probably not running. > [char...@test-zookeeper001 zookeeper-current]$ 2010-04-14 22:47:35,580 - DEBUG > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@1310] - ignoring > exception during input shutdown > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306) > at > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263) > at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609) > at > org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262) > > > If I connect with zkCli.sh, I can list the contents of zookeeper. If I make > changes to the schema on either of the other two nodes, test-zookeeper002 and > test-zookeeper003, both of which are running 3.2.1, the changes are reflected > on test-zookeeper001, which is running 3.3.0. > > When I exit zkCli.sh, however, zkServer.sh status starts flapping between > "Error contacting service. It is probably not running." and "Mode: follower", > as you can see below. > > Any ideas? I'd really rather not have to take the production zookeeper > cluster down to upgrade if it's not necessary. > > Thanks, > Charity. > > > > [char...@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status > JMX enabled by default > Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg > 2010-04-14 22:53:16,848 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioservercnxn$fact...@251] - > Accepted socket connection from /127.0.0.1:55284 > 2010-04-14 22:53:16,849 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@968] - Processing > stat command from /127.0.0.1:55284 > 2010-04-14 22:53:16,849 - WARN > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@606] - > EndOfStreamException: Unable to read additional data from client sessionid > 0x0, likely client has closed socket > 2010-04-14 22:53:16,850 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@1286] - Closed socket > connection for client /127.0.0.1:55284 (no session established for client) > Error contacting service. It is probably not running. > 2010-04-14 22:53:16,850 - DEBUG > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:nioserverc...@1310] - ignoring > exception during input shutdown > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306) > at > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263) > at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609) > at > org.apache.z
