I'm looking at the readonly mode code right now and it appears that the only way to set readonly mode is a global system property which means that the tests for this are only testing across 3 servers, all of which have readonly mode set. So, this MAY be a bug, but what a pain to figure out how to reproduce.
Global system properties: not even once. On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <[email protected]> wrote: > Not that I'm aware of... > > -Flavio > > > On 29 Jul 2016, at 17:24, Benjamin Jaton <[email protected]> > wrote: > > > > Hi Flavio, > > > > The bug / patch is only for the C client, does this issue affect the java > > client as well? > > > > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton < > [email protected]> > > wrote: > > > >> I will definitely try and I'll post an update, thanks again Flavio. > >> > >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <[email protected]> > wrote: > >> > >>> I think you're hitting this: > >>> > >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 < > >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466> > >>> > >>> which is resolved for 3.5.3 and trunk. You can try out the patch. > >>> > >>> -Flavio > >>> > >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <[email protected]> > >>> wrote: > >>>> > >>>> OK so I did another fresh install this time using 3.5.2-alpha, and the > >>> read > >>>> only client doesn't seem to try all the hosts available. > >>>> > >>>> I create my client with: > >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181, > >>>> 10.11.12.210:2181", 45000, this, true); > >>>> > >>>> In the following output, it only checks for 10.11.12.4:2181 and > >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where > >>> the > >>>> running server is. > >>>> > >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016 > >>>> 16:24 GMT > >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > >>> host.name > >>>> =w-rli09-ben > >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:java.version=1.8.0_11 > >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:java.vendor=Oracle Corporation > >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre > >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> > >>> > environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> > >>> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:java.io.tmpdir=/tmp > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:java.compiler=<NA> > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > >>> os.name > >>>> =Linux > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:os.arch=amd64 > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:os.version=3.8.0-44-generic > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > >>> user.name > >>>> =benji > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:user.home=/home/benji > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:os.memory.free=376MB > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:os.memory.max=5358MB > >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client > >>>> environment:os.memory.total=378MB > >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client > >>>> connection, connectString=10.11.9.70:2181 > >>>> <http://10.11.9.70:2181>,10.11.12.4:2181 > >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181> > >>>> sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6 > * > >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - > jute.maxbuffer > >>> is > >>>> 4194304 > >>>> Successfully established the connection with ZooKeeper > >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO > >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server > >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will > >>> not > >>>> attempt to authenticate using SASL (unknown error)* > >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN > >>> org.apache.zookeeper.ClientCnxn > >>>> - Session 0x0 for server null, unexpected error, closing socket > >>> connection > >>>> and attempting reconnect > >>>> java.net.ConnectException: Connection refused > >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > >>>> at > >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) > >>>> at > >>>> > >>> > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) > >>>> at > >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) > >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG > >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during > >>>> shutdown input > >>>> java.nio.channels.ClosedChannelException > >>>> at > >>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) > >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) > >>>> at > >>>> > >>> > org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) > >>>> at > >>>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) > >>>> at > >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) > >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG > >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during > >>>> shutdown output > >>>> java.nio.channels.ClosedChannelException > >>>> at > >>>> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) > >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) > >>>> at > >>>> > >>> > org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) > >>>> at > >>>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) > >>>> at > >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) > >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)] > >>> INFO > >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to server > >>>> 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181 > >. > >>>> Will not attempt to authenticate using SASL (unknown error)* > >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN > >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, > >>> unexpected > >>>> error, closing socket connection and attempting reconnect > >>>> java.net.ConnectException: Connection refused > >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > >>>> at > >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) > >>>> at > >>>> > >>> > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) > >>>> at > >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) > >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG > >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during > >>>> shutdown input > >>>> java.nio.channels.ClosedChannelException > >>>> at > >>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) > >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) > >>>> at > >>>> > >>> > org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) > >>>> at > >>>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) > >>>> at > >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) > >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG > >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during > >>>> shutdown output > >>>> java.nio.channels.ClosedChannelException > >>>> at > >>>> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) > >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) > >>>> at > >>>> > >>> > org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) > >>>> at > >>>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) > >>>> at > >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) > >>>> *Exception in thread "main" > >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: > >>>> KeeperErrorCode = ConnectionLoss for /* > >>>> at > >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > >>>> at > >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956) > >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985) > >>>> at > >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26) > >>>> at > >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13) > >>>> > >>>> > >>>> Does anybody else have issues with the readonly mode? > >>>> > >>>> Thanks > >>>> Benjamin > >>>> > >>>> > >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton < > >>> [email protected]> > >>>> wrote: > >>>> > >>>>> Very interesting, thank you Flavio. > >>>>> > >>>>> That should not happen because I have actually never started (even > >>> never > >>>>> created) the other servers. This server is the first one the client > >>> could > >>>>> have possibly ever seen. Maybe the server has to have been in a > >>> functioning > >>>>> ensemble at least once before being able to be read only able? > >>>>> I will recheck all my setup and do it again. > >>>>> Thanks again, > >>>>> Ben > >>>>> > >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <[email protected]> > >>> wrote: > >>>>> > >>>>>> From the logs, it looks like the server the client is trying to > >>> connect > >>>>>> to is behind compared to last zxid the client has seen: > >>>>>> > >>>>>> Refusing session request for client /10.11.12.4:49237 as it > >>> has > >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server > >>>>>> > >>>>>> I believe the behavior we currently have is that a client can > connect > >>> to > >>>>>> an RO server, but the server must have a view at least as recent as > >>> the > >>>>>> server. > >>>>>> > >>>>>> -Flavio > >>>>>> > >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <[email protected] > > > >>>>>> wrote: > >>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> I'm having problems using ZK in read-only mode. > >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the > >>>>>> installation to be in a 3 nodes ensemble. > >>>>>>> I start this node only (QA-E8WIN11) with > -Dreadonlymode.enabled=true. > >>>>>>> > >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly: > >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO > >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started > >>>>>>> > >>>>>>> Then I make a read-only enabled client: > >>>>>>> > >>>>>>> ZooKeeper zk = new > >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000, > >>> this, > >>>>>> true); > >>>>>>> System.out.println("Successfully established the connection > >>>>>> with ZooKeeper"); > >>>>>>> > >>>>>>> zk.getData("/", false, null); > >>>>>>> System.out.println("Done."); > >>>>>>> > >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not. > >>>>>>> > >>>>>>> When the ZK client picks QA-E8WIN11 first, it works. > >>>>>>> But when it picks a different node, it fails with: > >>>>>>> > >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client > >>>>>> connection, > >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181 > >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145 > >>>>>>> Successfully established the connection with ZooKeeper > >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket > >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 < > >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using > >>> SASL > >>>>>> (unknown error) > >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session > >>>>>> timed out, have not heard from server in 15001ms for sessionid 0x0, > >>> closing > >>>>>> socket connection and attempting reconnect > >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket > >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 < > >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using > >>> SASL > >>>>>> (unknown error) > >>>>>>> Exception in thread "main" > >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: > >>>>>> KeeperErrorCode = ConnectionLoss for / > >>>>>>> at > >>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > >>>>>>> at > >>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753) > >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782) > >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26) > >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16) > >>>>>>> > >>>>>>> I'm attaching all the logs. The server and the client are on the > same > >>>>>> machine QA-E8WIN11. > >>>>>>> > >>>>>>> Any help would be greatly appreciated! > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Benjamin Jaton > >>>>>>> <zkclient.log><zkserver.log> > >>>>>> > >>>>>> > >>>>> > >>> > >>> > >> > >
