Thanks, Manikumar for pointing the typo. Upon setting the ACL rules are told by you above, it worked fine. I was able to run secure (on external interface) and insecure (on internal interface) mode.
As a reference to others in the forum, in addition to the server.properties that I have posted above, these are the two things I did: 1. Changed super.users to *super.users=User:CN=Kafka1;UserANONYMOUS2. Used the ACL rule as told by Manikumar:bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:ANONYMOUS --allow-host \* --operation Read --topic test* Many thanks Manikumar. Thanks. On Wed, Apr 4, 2018 at 7:25 PM, Manikumar <manikumar.re...@gmail.com> wrote: > Hi, > > User name is ANONYMOUS, not CN=ANONYMOUS. You can enable authorizer logs > (kafka-authorizer.log) and check for any deny errors. > supers.users can be configured with same value across all brokers. > > sh kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 > --add --allow-principal User:ANONYMOUS --allow-host \* --operation Read > --topic test > > On Thu, Apr 5, 2018 at 2:39 AM, Darshan <purandare.dars...@gmail.com> > wrote: > >> Hi Manikumar >> >> I pushed ACLs for User:ANONYMOUS and when I list them they are listed as >> shown. Can you please suggest if server.properties needs a change ? >> >> *[alpha: user@Kafka1 kafka_2.12-0.10.2.1]$ bin/kafka-acls.sh >> --authorizer-properties zookeeper.connect=Kafka1-1:2181 --list --topic >> topic05* >> *Current ACLs for resource `Topic:topic05`:* >> * User:CN=ANONYMOUS has Allow permission for operations: Describe >> from hosts: ** >> * User:CN=Producer05 has Allow permission for operations: Write >> from hosts: ** >> * User:CN=Producer05 has Allow permission for operations: Describe >> from hosts: ** >> * User:CN=ANONYMOUS has Allow permission for operations: Write >> from hosts: ** >> >> When I start sending messages using kafka console client, I see the >> following error. topic05 is created with 16 partitions, but no directory is >> created in logs directory. Can someone please help ? >> >> *[alpha: user@Kafka1 kafka_2.12-0.10.2.1]$ bin/kafka-console-producer.sh >> --broker-list Kafka1:9092 --topic topic05* >> *WorldHello* >> *[2018-04-04 21:01:40,338] WARN Error while fetching metadata with >> correlation id 1 : {topic05=UNKNOWN_TOPIC_OR_PARTITION} >> (org.apache.kafka.clients.NetworkClient)* >> *[2018-04-04 21:01:40,437] WARN Error while fetching metadata with >> correlation id 2 : {topic05=UNKNOWN_TOPIC_OR_PARTITION} >> (org.apache.kafka.clients.NetworkClient)* >> *[2018-04-04 21:01:40,538] WARN Error while fetching metadata with >> correlation id 3 : {topic05=UNKNOWN_TOPIC_OR_PARTITION} >> (org.apache.kafka.clients.NetworkClient)* >> *[2018-04-04 21:01:40,640] WARN Error while fetching metadata with >> correlation id 4 : {topic05=UNKNOWN_TOPIC_OR_PARTITION} >> (org.apache.kafka.clients.NetworkClient)* >> *[2018-04-04 21:01:40,740] WARN Error while fetching metadata with >> correlation id 5 : {topic05=UNKNOWN_TOPIC_OR_PARTITION} >> (org.apache.kafka.clients.NetworkClient)* >> *[2018-04-04 21:01:40,841] WARN Error while fetching metadata with >> correlation id 6 : {topic05=UNKNOWN_TOPIC_OR_PARTITION} >> (org.apache.kafka.clients.NetworkClient)* >> *[2018-04-04 21:01:40,943] WARN Error while fetching metadata with >> correlation id 7 : {topic05=UNKNOWN_TOPIC_OR_PARTITION} >> (org.apache.kafka.clients.NetworkClient)* >> *[2018-04-04 21:01:41,045] WARN Error while fetching metadata with >> correlation id 8 : {topic05=UNKNOWN_TOPIC_OR_PARTITION} >> (org.apache.kafka.clients.NetworkClient)* >> >> >> >> >> >> >> >> *server.properties for our Kafka-1,2,3. They are identical >> except broker.id <http://broker.id/> and super.users properties.# ID and >> basic topic creationbroker.id >> <http://broker.id/>=1auto.create.topics.enable=truedelete.topic.enable=true# >> LISTERN Settingslisteners=INTERNAL://1.1.1.165:9092 >> <http://1.1.1.165:9092/>,EXTERNAL://172.21.190.176:9093 >> <http://172.21.190.176:9093/>advertised.listeners=INTERNAL://1.1.1.165:9092 >> <http://1.1.1.165:9092/>,EXTERNAL://172.21.190.176:9093 >> <http://172.21.190.176:9093/>listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSLinter.broker.listener.name >> <http://inter.broker.listener.name/>=INTERNALhost.name >> <http://host.name/>=172.21.190.176# Security >> Settingsssl.keystore.location=keystore.jksssl.keystore.password=passwordssl.key.password=passwordssl.truststore.location=truststore.jksssl.truststore.password=passwordssl.keystore.type=JKSssl.truststore.type=JKSsecurity.protocol=SSLssl.client.auth=requiredallow.everyone.if.no.acl.found=falseauthorizer.class.name >> <http://authorizer.class.name/>=kafka.se >> <http://kafka.se>curity.auth.SimpleAclAuthorizersuper.users=User:CN=Kafka1* >> >> >> On Tue, Apr 3, 2018 at 10:42 PM, Manikumar <manikumar.re...@gmail.com> >> wrote: >> >>> @Darshan, >>> For PLAINTEXT channels, principal will be "ANONYMOUS". You need to give >>> produce/consume permissions >>> to "User:ANONYMOUS" >>> >>> >>> On Wed, Apr 4, 2018 at 8:10 AM, Joe Hammerman < >>> jhammer...@squarespace.com.invalid> wrote: >>> >>> > Hi all, >>> > >>> > Is it possible to run mixed mode with PLAINTEXT and SSL with no SASL? >>> What >>> > should port and advertised listeners values in kafka-server.properties >>> be >>> > set in order to configure such an access profile? We want to so in >>> order to >>> > be able to perform healthchecking on the loopback device without >>> having to >>> > negotiate an SSL connection. >>> > >>> > Thanks in advance for any assistance anyone can provide, >>> > Joseph Hammerman >>> > >>> > On Tue, Apr 3, 2018 at 8:06 PM, Martin Gainty <mgai...@hotmail.com> >>> wrote: >>> > >>> > > (from MessageBroker consumer) >>> > > >>> > > >tracert MessageProducer >>> > > >>> > > if zk server is found in tracert >>> > > >>> > > then yes you have a MB quorom >>> > > >>> > > FWIR Mixing PLAINTEXT with SASL-SSL on ZK is not supported >>> > > https://stackoverflow.com/questions/46912937/is-it- >>> > > possible-to-connect-zookeeper-and-kafka-via-sasl-kafka-broke >>> r-and-its-cl >>> > > >>> > > but there is a ZK SASL SASL-Auth exception backdoor in >>> > > zookeeper.properties when >>> > > zookeeper.main_connection_despite_sasl_failure="yes" >>> > > >>> > > >>> > > catch (SaslException e) { >>> > > LOG.warn("Client failed to SASL authenticate: " + e); >>> > > if ((System.getProperty("zookeeper.maintain_connection_ >>> > despite_sasl_failure") >>> > > != null) >>> > > && >>> > > (System.getProperty("zookeeper.maintain_connection_ >>> > > despite_sasl_failure").equals("yes"))) { >>> > > LOG.warn("Maintaining client connection despite SASL >>> authentication >>> > > failure."); >>> > > } else { >>> > > LOG.warn("Closing client connection due to SASL authentication >>> > > failure."); >>> > > cnxn.close(); >>> > > .. >>> > > } >>> > > >>> > > >>> > > does this conform to your findings? >>> > > >>> > > M >>> > > ______________________________________________ >>> > > >>> > > >>> > > >>> > > ________________________________ >>> > > From: Darshan <purandare.dars...@gmail.com> >>> > > Sent: Tuesday, April 3, 2018 6:47 PM >>> > > To: users@kafka.apache.org >>> > > Subject: Re: advertised.listeners >>> > > >>> > > We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from >>> this >>> > > piece of info ? Thanks. >>> > > >>> > > On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mgai...@hotmail.com> >>> > wrote: >>> > > >>> > > > tracert MessageBrokerIP >>> > > > >>> > > > do you see ZK server in the trace? >>> > > > >>> > > > if yes then you are running kafka-cluster >>> > > > >>> > > > (ZK does not support mixed mode but there is a backdoor >>> > > > zookeeper.properties config attribute that allows plaintext >>> clients to >>> > > > bypass sasl auth) >>> > > > >>> > > > ? >>> > > > >>> > > > Martin >>> > > > ______________________________________________ >>> > > > >>> > > > >>> > > > >>> > > > ________________________________ >>> > > > From: Darshan <purandare.dars...@gmail.com> >>> > > > Sent: Tuesday, April 3, 2018 5:45 PM >>> > > > To: rajinisiva...@gmail.com >>> > > > Cc: users@kafka.apache.org >>> > > > Subject: Re: advertised.listeners >>> > > > >>> > > > Hi Rajini >>> > > > >>> > > > The above configuration that you mentioned a while back helped me >>> sort >>> > > the >>> > > > issue of listeners and I was also able to run Kafka 0.10.2.1 with >>> SSL >>> > and >>> > > > ACLs as well from one of your other posts. >>> > > > >>> > > > I wanted to ask you if it is possible to run Kafka in a mixed >>> security >>> > > > mode? i.e. external producers who are on 172.x.x.x interface can >>> use >>> > the >>> > > > SSL to send/receive data from/to our Kafka brokers, but our >>> internal >>> > > > consumer can read/write on PLAINTEXT channel to Kafka. >>> > > > >>> > > > Here is my server.properties, but my internal producer which does >>> not >>> > > have >>> > > > any keystore and truststore is getting the following error: >>> > > > *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with >>> > > > correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED} >>> > > > (org.apache.kafka.clients.NetworkClient)* >>> > > > >>> > > > >>> > > > *server.properties for our Kafka-1,2,3. They are identical except >>> > > > broker.id >>> > > > <http://broker.id> and super.users properties.* >>> > > broker.id - This website is for sale! - broker Resources and >>> > > Information.<http://broker.id/> >>> > > broker.id >>> > > This website is for sale! broker.id is your first and best source >>> for >>> > all >>> > > of the information you’re looking for. From general topics to more of >>> > what >>> > > you would expect to find here, broker.id has it all. We hope you >>> find >>> > > what you are searching for! >>> > > >>> > > >>> > > >>> > > > broker.id - This website is for sale! - broker Resources >>> and >>> > > > Information.<http://broker.id/> >>> > > > broker.id >>> > > > This website is for sale! broker.id is your first and best source >>> for >>> > > all >>> > > > of the information you’re looking for. From general topics to more >>> of >>> > > what >>> > > > you would expect to find here, broker.id has it all. We hope you >>> find >>> > > > what you are searching for! >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > # ID and basic topic creation >>> > > > broker.id=1 >>> > > > auto.create.topics.enable=true >>> > > > delete.topic.enable=true >>> > > > >>> > > > # LISTERN Settings >>> > > > listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093 >>> > > > advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL:// >>> > > > 172.21.190.176:9093 >>> > > > listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL >>> > > > inter.broker.listener.name=INTERNAL >>> > > > host.name=172.21.190.176 >>> > > > >>> > > > # Security Settings >>> > > > ssl.keystore.location=keystore.jks >>> > > > ssl.keystore.password=password >>> > > > ssl.key.password=password >>> > > > ssl.truststore.location=truststore.jks >>> > > > ssl.truststore.password=password >>> > > > ssl.keystore.type=JKS >>> > > > ssl.truststore.type=JKS >>> > > > security.protocol=SSL >>> > > > ssl.client.auth=required >>> > > > allow.everyone.if.no.acl.found=false >>> > > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer >>> > > > super.users=User:CN=Kafka1 >>> > > > >>> > > > Can you please point out if anything needs to be modified ? >>> > > > >>> > > > Many thanks. >>> > > > >>> > > > --Darshan >>> > > > >>> > > > On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram < >>> > > rajinisiva...@gmail.com> >>> > > > wrote: >>> > > > >>> > > > > If you want to use different interfaces with the same security >>> > > protocol, >>> > > > > you can specify listener names. You can then also configure >>> different >>> > > > > security properties for internal/external if you need. >>> > > > > >>> > > > > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093 >>> > > > > >>> > > > > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172. >>> > x.x.x:9093 >>> > > > > >>> > > > > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL >>> > > > > >>> > > > > inter.broker.listener.name=INTERNAL >>> > > > > >>> > > > > On Wed, May 31, 2017 at 6:22 PM, Raghav <raghavas...@gmail.com> >>> > wrote: >>> > > > > >>> > > > > > Hello Darshan >>> > > > > > >>> > > > > > Have you tried SSL://0.0.0.0:9093 ? >>> > > > > > >>> > > > > > Rajani had suggested something similar to me a week back while >>> I >>> > was >>> > > > > > trying to get a ACL based setup. >>> > > > > > >>> > > > > > Thanks. >>> > > > > > >>> > > > > > On Wed, May 31, 2017 at 8:58 AM, Darshan < >>> > > purandare.dars...@gmail.com> >>> > > > > > wrote: >>> > > > > > >>> > > > > >> Hi >>> > > > > >> >>> > > > > >> Our Kafka broker has two IPs on two different interfaces. >>> > > > > >> >>> > > > > >> eth0 has 172.x.x.x for external leg >>> > > > > >> eth1 has 1.x.x.x for internal leg >>> > > > > >> >>> > > > > >> >>> > > > > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is >>> on >>> > > > 1.x.x.x >>> > > > > >> subnet. >>> > > > > >> >>> > > > > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then >>> > Producer >>> > > > can >>> > > > > >> producer the message, but Consumer cannot receive the message. >>> > > > > >> >>> > > > > >> What value should we use for advertised.listeners so that >>> Producer >>> > > can >>> > > > > >> write and Consumers can read ? >>> > > > > >> >>> > > > > >> Thanks. >>> > > > > >> >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > -- >>> > > > > > Raghav >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> >> >> >