Re: question on ZAB protocol
>> so the client and the cluster has an inconsistent view. I would be reluctant to conclude this is an inconsistent view as a client should always consult server to get the latest state, rather than derive the state from the response of the request, which is not reliable if the request "fails" as one never know the request truly fails or not in cases such as connection loss. Though, when the request succeeds - that ZooKeeper guarantees the write is persistent to quorum. On Sun, Feb 16, 2020 at 5:26 AM wrote: > Sorry for misunderstood. I think the client could not receive the 'error' > message or even if it receive any ack, the ack should be 'undeterminated' > > 发自我的 iPhone > > > 在 2020年2月16日，10:35，jonefeewang 写道： > > > > Norbert Kalmar-2 wrote > >> Hi, > >> > >> A would not have confirmed in this case to the client the write. Sending > >> ACK means the followers have written the transaction to disc. Leader (in > >> this case A) still needs to send COMMIT message to the followers. > >> It goes like this: > >> - LEADER(A) receives a write, so it creates a transaction and send it to > >> all FOLLOWERs. > >> - FOLLOWERs receive the transaction and writes it to disc (txnlog). It > >> does > >> NOT apply to the datatree. > >> - After writing to disc FOLLOWERs send ACK to LEADER(A) (Nothing at this > >> point is acknowledged to the client) > >> - After LEADER(A) receives quorum of ACK, then, and only then will it > >> apply > >> to the datatree and send COMMIT message to all FOLLOWERs to do the same. > >> And also ACK to client that the write is complete. And at this point the > >> data sent by the client is saved in the txnlogs of the quorum. > >> > >> Hope this helps, > >> > >> Regards, > >> Norbert > >> > >> On Sat, Feb 15, 2020 at 5:20 AM > > > >> hnwyllmm@ > > > >> wrote: > >> > >>> How do you know A has sent the ack to client before he die ? > >>> > >>> 发自我的 iPhone > >>> > 在 2020年2月15日，09:15，jonefeewang > > > >> jonefeewang@ > > > >> 写道： > > I also have the same question like this below: > > > let's say we have nodes A B C D E, now A is the leader > > A broadcasts <1,1>, it reaches B, then A, B die, C D E elect someone, > the new system is going to throw away <1,1> since it does not know its > existence, right? > > start from scratch, > A broadcasts<1,1> , it reaches all, all send ACK to A, but A dies > before receiving the ACK, then BCDE elects someone, and the new leader > sees <1,1> in log, so it broadcasts <1,1> to BCDE, which all commit > it. now if we look back, when A dies, the client should get a "write > failure", but now after BCDE relection, the written value does get > into the system ??? the client and the cluster has an inconsistent > view > >>> ?? > > > > > > -- > Sent from: http://zookeeper-user.578899.n2.nabble.com/ > >>> > >>> > > > > > > Sorry, I think I need to make the question more clear : > > > > 1. A broadcasts<1,1> , it reaches all, all send ACK to A > > 2. A dies before receiving the ACK, > > 3. BCDE elects someone, and the new leader sees <1,1> in log, so it > > broadcasts <1,1> to BCDE, which all commit it. > > > > now if we look back, when A dies, the client should get a "write > > failure", but now after BCDE relection, the written value does get into > the > > system 。 > > > > so in the last, the client got a write error(probably think this write > did > > not succeed), but the server clusters did write this value in their log > and > > datatree. > > > > so the client and the cluster has an inconsistent view. > > > > > > > > > > -- > > Sent from: http://zookeeper-user.578899.n2.nabble.com/ > >
Re: Enabling Auth between Zookeeper Servers
Hello, I think I found the issue... One can't use the same username for clients and quorums. I configured all of them to be "zookeeper", but in the server-part of the jaas.conf it should probably be more like "kafka" as it's Kafka which authenticates to the zookeeper in that case and zookeepers are using the qorum-part to authenticate to each other. Correct? If that's correct the exception message is completely wrong. It can find the file, it can read the file and it even finds the server-part, but the server-part itself has wrong configuration. At least with the hanged username in the server-part I got a new exception: 2020-02-17 19:28:17,994 [myid:1] - ERROR [main:ZooKeeperServerMain@83] - Unexpected exception, exiting abnormally java.io.IOException: No snapshot found, but there are log entries. Something is broken! Which was probably caused by non-cleaned folders of some previous deployments. So I added the "snapshot.trust.empty=true" to the config to have it start and rebuild the snapshot. And now my zookeeper is running just fine! :) @Mate: as I copied the jaas.conf from your repo is that the exact file you used for testing? Because changing the "user_zookeeper" to "user_kafka" in the server-part fixed it. My next task now is to get Kafka authenticated to zookeeper and get ACLs working. Will be fun :) And I should probably create a ticket to get the jaas.conf-error message fixed!? Best regards Sebastian On 17-Feb-20 1:50 PM, Sebastian Schmitz wrote: Hey, I also just tried using 3.5.7, but same problem... Best regards Sebastian On 17-Feb-20 11:34 AM, Sebastian Schmitz wrote: Hi Mate, that's what I also tried. I copied it to the /opt/zookeeper-cluster/-folder and got the same exception just with the new path. So, if that config works on your side it might be my environment then!? Maybe it's a problem with the base-image openjdk:11-jre-stretch which I use for the container... I'll try using the openjdk:8u222-jre you're using. Best regards Sebastian On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote: Hi Sebastian, It's strange indeed... I also see the owner is root. That should work in docker usually, given that you run the zookeeper process with the root user. Maybe copying it to a different folder? I see that the conf folder has different owner, maybe the java security library doesn't like that? But honestly, I don't have any useful explanation. Good luck! Mate On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: Hey Mate, now it gets really weird. I get the file not found exception: '.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] - No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf java.lang.SecurityException: java.io.IOException: /opt/zookeeper-cluster/zookeeper/conf/jaas.conf (No such file or directory) at java.base/sun.security.provider.ConfigFile$Spi.(Unknown Source) at java.base/sun.security.provider.ConfigFile.(Unknown Source) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at java.base/java.lang.Class.newInstance(Unknown Source) at java.base/javax.security.auth.login.Configuration$2.run(Unknown Source) at java.base/javax.security.auth.login.Configuration$2.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown Source) at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) Caused by: java.io.IOException: /opt/zookeeper-cluster/zookeeper/conf/jaas.conf (No such file or directory) at java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown Source) at java.base/sun.security.provider.ConfigFile$Spi.init(Unknown Source) ... 18 more 2020-02-16 18:27:50,566
