Thanks for the tips to both! Still struggling with this.
If I got it right, issue it seems is in different response of ZooKeeper server (3.4.6) for same request - when authentication is on or off, ZooKeeper responds differently when trying to create a node that already exists. Not using Kerberos, using DIGEST-MD5 as SASL mechanism. I'm trying to enable just ZooKeeper authentication, between Kafka brokers and ZooKeeper, and an app and ZooKeeper. InvalidACL is not thrown and everything works OK, without authentication enabled. App has logic to explicitly create topic using AdminUtils.createTopic with custom topic configuration. That fails now with InvalidACL. No custom ACL is being used in the calls. What I can see from stack trace and Kafka source code, when trying to write topic configuration data https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala#L436, underlying ZkUtils call first gets NoNode for the topic config path "/config/topics/foo" and then tries to create parent and that fails with InvalidACL https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L424 Will check with Kafka community, why does the AdminUtils, when tying to update topic config node data assumes that it got NoNode because parent node doesn't exist - checked manually and /config/topics exists (likely broker creates it on startup) and has ACLs set (world can read, and Kafka's user can cdrwa). App trying to create the topic explicitly has same JAAS config file as Kafka broker, and zkUtils passed to AdminUtils.createTopic has isSecure set to true, so it shouldn't be permissions or authentication issue. Without authentication, same ZkUtils call to create parent fails with ZkNodeExistsException but that exception is swallowed by ZkClient (see https://github.com/sgroschupf/zkclient/blob/master/src/main/java/org/I0Itec/zkclient/ZkClient.java#L295 ) so ZkUtils never sees it. While it seems with ZooKeeper 3.4.6 server and authentication on, client trying to create a node that already exists gets InvalidACL, so topic creation fails - if true, is this inconsistency in response a feature or a bug in ZooKeeper? Kind regards, Stevo Slavic. On Sat, Jul 2, 2016 at 2:40 AM, Raúl Gutiérrez Segalés <[email protected]> wrote: > Another way to debug this is by using zk-dump [0] to dump requests/replies > and see if the expected ACL is being send across the wire. > > -rgs > > [0] https://github.com/twitter/zktraffic > On Jul 1, 2016 7:03 AM, "Flavio Junqueira" <[email protected]> wrote: > > > Hi Stevo, > > > > The logs indicate whether it has connected and authorized properly. Make > > sure your keytab file is all good. > > > > One other suggestion is to try with the zkCli against the ensemble before > > you try to run Kafka to make sure that your configuration is correct. > > > > -Flavio > > > > > On 01 Jul 2016, at 14:48, Stevo Slavić <[email protected]> wrote: > > > > > > Hello Apache ZooKeeper community, > > > > > > As subject states, I'm having difficulties determining what could be > > > causing InvalidACL to be thrown, any help would be appreciated. > > > > > > Kafka's ZkUtils I assume works OK and sends default ACLs with create > path > > > request > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L80 > > > but server replies with InvalidACL error. Even at trace log level I > > cannot > > > see any useful information in the console. > > > > > > Kind regards, > > > Stevo Slavic. > > > > >
