( I have 3 test nodes) get /brokers/ids/11
{"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL:// n1.test.com:9093 "],"jmx_port":-1,"host":null,"timestamp":"1502310695312","port":-1,"version":4} cZxid = 0x40002787d ctime = Thu Aug 10 04:31:37 HKT 2017 mZxid = 0x40002787d mtime = Thu Aug 10 04:31:37 HKT 2017 pZxid = 0x40002787d cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x35d885c689c00a6 dataLength = 168 numChildren = 0 get /brokers/ids/12 {"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL:// n2.test.com:9093 "],"jmx_port":-1,"host":null,"timestamp":"1502284073115","port":-1,"version":4} cZxid = 0x400026c66 ctime = Wed Aug 09 21:07:53 HKT 2017 mZxid = 0x400026c66 mtime = Wed Aug 09 21:07:53 HKT 2017 pZxid = 0x400026c66 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x25d6b41469a0110 dataLength = 168 numChildren = 0 get /brokers/ids/13 {"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL:// n3.test.com:9093 "],"jmx_port":-1,"host":null,"timestamp":"1502284080461","port":-1,"version":4} cZxid = 0x400026c6c ctime = Wed Aug 09 21:07:59 HKT 2017 mZxid = 0x400026c6c mtime = Wed Aug 09 21:07:59 HKT 2017 pZxid = 0x400026c6c cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x35d885c689c00a2 dataLength = 168 numChildren = 0 On Thu, Aug 10, 2017 at 5:03 AM, Ascot Moss <ascot.m...@gmail.com> wrote: > > About: > zookeeper-shell.sh localhost:2181 > get /brokers/ids/11 > > > The result: > > zookeeper-shell.sh n1.test.com:2181 > > Connecting to n1.test.com:2181 > > Welcome to ZooKeeper! > > JLine support is disabled > > WATCHER:: > > WatchedEvent state:SyncConnected type:None path:null > > WATCHER:: > > > > > get /brokers/ids/11 > > WatchedEvent state:SaslAuthenticated type:None path:null > > {"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL:// > n1.test.com:9093"],"jmx_port":-1,"host":null,"timestamp":"1502310695312"," > port":-1,"version":4} > > cZxid = 0x40002787d > > ctime = Thu Aug 10 04:31:37 HKT 2017 > > mZxid = 0x40002787d > > mtime = Thu Aug 10 04:31:37 HKT 2017 > > pZxid = 0x40002787d > > cversion = 0 > > dataVersion = 0 > > aclVersion = 0 > > ephemeralOwner = 0x35d885c689c00a6 > > dataLength = 168 > > numChildren = 0 > > On Thu, Aug 10, 2017 at 4:46 AM, Ascot Moss <ascot.m...@gmail.com> wrote: > >> About: zookeeper-shell.sh localhost:2181 >> get /brokers/ids/11 >> >> The result: >> >> zookeeper-shell.sh n1.test.com:2181 >> >> Connecting to n1.test.com:2181 >> >> Welcome to ZooKeeper! >> >> JLine support is disabled >> >> WATCHER:: >> >> WatchedEvent state:SyncConnected type:None path:null >> >> WATCHER:: >> >> WatchedEvent state:SaslAuthenticated type:None path:null >> >> >> On Thu, Aug 10, 2017 at 4:43 AM, Ascot Moss <ascot.m...@gmail.com> wrote: >> >>> FYI, about zookeeper, I used my existing zookeeper (as I have existing >>> zookeeper up and running, which is also used for hbase) >>> >>> zookeeper versoom: 3.4.10 >>> >>> zoo.cfg >>> ###### >>> >>> tickTime=2000 >>> >>> initLimit=10 >>> >>> syncLimit=5 >>> >>> dataDir=/usr/local/zookeeper/data >>> >>> dataLogDir=/usr/local/zookeeper/datalog >>> >>> clientPort=2181 >>> >>> maxClientCnxns=60 >>> >>> server.1=n1.test.com:2888:3888 >>> >>> server.2=n2.test.com:2888:3888 >>> >>> server.3=n3.test.com:2888:3888 >>> >>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenti >>> cationProvider >>> >>> jaasLoginRenew=3600000 >>> >>> requireClientAuthScheme=sasl >>> >>> zookeeper.allowSaslFailedClients=false >>> >>> kerberos.removeHostFromPrincipal=true >>> >>> ###### >>> >>> >>> >>> On Thu, Aug 10, 2017 at 4:35 AM, Ascot Moss <ascot.m...@gmail.com> >>> wrote: >>> >>>> server.properties >>>> >>>> ###### >>>> >>>> broker.id=11 >>>> >>>> port=9093 >>>> >>>> host.name=n1 >>>> >>>> advertised.host.name=192.168.0.11 >>>> >>>> allow.everyone.if.no.acl.found=true >>>> >>>> super.users=User:CN=n1.test.com,OU=TEST,O=TEST,L=TEST,ST=TEST,C=TEST >>>> >>>> listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/> >>>> >>>> advertised.listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/> >>>> >>>> ssl.client.auth=required >>>> >>>> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 >>>> >>>> ssl.keystore.type=JKS >>>> >>>> ssl.truststore.type=JKS >>>> >>>> security.inter.broker.protocol=SSL >>>> >>>> ssl.keystore.location=/home/kafka/kafka.server.keystore.jks >>>> >>>> ssl.keystore.password=Test2017 >>>> >>>> ssl.key.password=Test2017 >>>> >>>> ssl.truststore.location=/home/kafka/kafka.server.truststore.jks >>>> >>>> ssl.truststore.password=Test2017 >>>> >>>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer >>>> >>>> principal.builder.class=org.apache.kafka.common.security.aut >>>> h.DefaultPrincipalBuilder >>>> >>>> num.replica.fetchers=4 >>>> >>>> replica.fetch.max.bytes=1048576 >>>> >>>> replica.fetch.wait.max.ms=500 >>>> >>>> replica.high.watermark.checkpoint.interval.ms=5000 >>>> >>>> replica.socket.timeout.ms=30000 >>>> >>>> replica.socket.receive.buffer.bytes=65536 >>>> >>>> replica.lag.time.max.ms=10000 >>>> >>>> controller.socket.timeout.ms=30000 >>>> >>>> controller.message.queue.size=10 >>>> >>>> default.replication.factor=3 >>>> >>>> log.dirs=/usr/log/kafka >>>> >>>> kafka.logs.dir=/usr/log/kafka >>>> >>>> num.partitions=20 >>>> >>>> message.max.bytes=1000000 >>>> >>>> auto.create.topics.enable=true >>>> >>>> log.index.interval.bytes=4096 >>>> >>>> log.index.size.max.bytes=10485760 >>>> >>>> log.retention.hours=720 >>>> >>>> log.flush.interval.ms=10000 >>>> >>>> log.flush.interval.messages=20000 >>>> >>>> log.flush.scheduler.interval.ms=2000 >>>> >>>> log.roll.hours=168 >>>> >>>> log.retention.check.interval.ms=300000 >>>> >>>> log.segment.bytes=1073741824 >>>> >>>> delete.topic.enable=true >>>> >>>> socket.request.max.bytes=104857600 >>>> >>>> socket.receive.buffer.bytes=1048576 >>>> >>>> socket.send.buffer.bytes=1048576 >>>> >>>> num.io.threads=8 >>>> >>>> num.network.threads=8 >>>> >>>> queued.max.requests=16 >>>> >>>> fetch.purgatory.purge.interval.requests=100 >>>> >>>> producer.purgatory.purge.interval.requests=100 >>>> >>>> zookeeper.connect=n1:2181,n2:2181,n3:2181 >>>> >>>> zookeeper.connection.timeout.ms=2000 >>>> >>>> zookeeper.sync.time.ms=2000 >>>> >>>> ###### >>>> >>>> >>>> >>>> >>>> >>>> producer.properties >>>> >>>> ###### >>>> >>>> bootstrap.servers=n1.test.com:9093 <http://n1.test.com:9092/> >>>> >>>> security.protocol=SSL >>>> >>>> ssl.truststore.location=/home/kafka/kafka.client.truststore.jks >>>> >>>> ssl.truststore.password=testkafka >>>> >>>> ssl.keystore.location=/home/kafka/kafka.client.keystore.jks >>>> >>>> ssl.keystore.password=testkafka >>>> >>>> ssl.key.password=testkafka >>>> ##### >>>> >>>> >>>> (I had tried to switch to another port, 9093 is the correct port) >>>> >>>> On Thu, Aug 10, 2017 at 4:28 AM, M. Manna <manme...@gmail.com> wrote: >>>> >>>>> Your openssl test is showing connected with port 9092. but your >>>>> previous >>>>> messages show 9093 - is there some typo issues? Where is SSL running >>>>> >>>>> Please share the following and don't leave any details out. This will >>>>> only >>>>> create more assumptions. >>>>> >>>>> 1) server.properties >>>>> 2) Zookeeper.properties >>>>> >>>>> Also, run the following command (when the cluster is running) >>>>> zookeeper-shell.sh localhost:2181 >>>>> get /brokers/ids/11 >>>>> >>>>> Does it show that your broker #11 is connected? >>>>> >>>>> >>>>> >>>>> >>>>> On 9 August 2017 at 21:17, Ascot Moss <ascot.m...@gmail.com> wrote: >>>>> >>>>> > Dear Manna, >>>>> > >>>>> > >>>>> > What's the status of your SSL? Have you verified that the setup is >>>>> working? >>>>> > Yes, I used " >>>>> > >>>>> > openssl s_client -debug -connect n1.test.com:9092 -tls1 >>>>> > Output: >>>>> > >>>>> > CONNECTED(00000003) >>>>> > >>>>> > write to 0x853e70 [0x89fd43] (155 bytes => 155 (0x9B)) >>>>> > >>>>> > 0000 - 16 03 01 00 96 01 00 00-92 03 01 59 8b 6d 0d b1 >>>>> ...........Y.m.. >>>>> > ... >>>>> > >>>>> > Server certificate >>>>> > >>>>> > -----BEGIN CERTIFICATE----- >>>>> > >>>>> > CwwCSEsxGT............ >>>>> > >>>>> > -----END CERTIFICATE----- >>>>> > >>>>> > --- >>>>> > >>>>> > SSL handshake has read 2470 bytes and written 161 bytes >>>>> > >>>>> > --- >>>>> > >>>>> > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA >>>>> > >>>>> > PSK identity hint: None >>>>> > >>>>> > Start Time: 1502309645 >>>>> > >>>>> > Timeout : 7200 (sec) >>>>> > >>>>> > Verify return code: 19 (self signed certificate in certificate >>>>> chain) >>>>> > >>>>> > --- >>>>> > >>>>> > Regards >>>>> > >>>>> > On Wed, Aug 9, 2017 at 10:29 PM, M. Manna <manme...@gmail.com> >>>>> wrote: >>>>> > >>>>> > > Hi, >>>>> > > >>>>> > > What's the status of your SSL? Have you verified that the setup is >>>>> > working? >>>>> > > >>>>> > > You can enable rough logins using log4j.properties file supplier >>>>> with >>>>> > kafka >>>>> > > and set the root logging level to DEBUG. This prints out more info >>>>> to >>>>> > trace >>>>> > > things. Also, you can enable security logging by adding >>>>> > > -Djavax.security.debug=all >>>>> > > >>>>> > > Please share your producer/broker configs with us. >>>>> > > >>>>> > > Kindest Regards, >>>>> > > M. Manna >>>>> > > >>>>> > > On 9 August 2017 at 14:38, Ascot Moss <ascot.m...@gmail.com> >>>>> wrote: >>>>> > > >>>>> > > > Hi, >>>>> > > > >>>>> > > > >>>>> > > > I have setup Kafka 0.10.2.1 with SSL. >>>>> > > > >>>>> > > > >>>>> > > > Check Status: >>>>> > > > >>>>> > > > openssl s_client -debug -connect n1:9093 -tls1 >>>>> > > > >>>>> > > > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA >>>>> > > > >>>>> > > > ... SSL-Session: >>>>> > > > >>>>> > > > Protocol : TLSv1 >>>>> > > > >>>>> > > > PSK identity hint: None >>>>> > > > >>>>> > > > Start Time: 1502285690 >>>>> > > > >>>>> > > > Timeout : 7200 (sec) >>>>> > > > >>>>> > > > Verify return code: 19 (self signed certificate in >>>>> certificate >>>>> > chain) >>>>> > > > >>>>> > > > >>>>> > > > Create Topic: >>>>> > > > >>>>> > > > kafka-topics.sh --create --zookeeper n1:2181,n2:2181,n3:2181 >>>>> > > > --replication-factor 3 --partitions 3 --topic test02 >>>>> > > > >>>>> > > > ERROR [ReplicaFetcherThread-2-111], Error for partition >>>>> [test02,2] to >>>>> > > > broker 1:org.apache.kafka.common.erro >>>>> rs.UnknownTopicOrPartitionExcepti >>>>> > > on: >>>>> > > > This server does not host this topic-partition. >>>>> > > > (kafka.server.ReplicaFetcherThread) >>>>> > > > >>>>> > > > However, if I run describe topic, I can see it is created >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > Describe Topic: >>>>> > > > >>>>> > > > kafka-topics.sh --zookeeper n1:2181,n2:2181,n3:2181 --describe >>>>> --topic >>>>> > > > test02 >>>>> > > > >>>>> > > > Topic:test02 PartitionCount:3 ReplicationFactor:3 Configs: >>>>> > > > >>>>> > > > Topic: test02 Partition: 0 Leader: 12 Replicas: 12,13,11 Isr: >>>>> 12,13,11 >>>>> > > > >>>>> > > > Topic: test02 Partition: 1 Leader: 13 Replicas: 13,11,12 Isr: >>>>> 13,11,12 >>>>> > > > >>>>> > > > Topic: test02 Partition: 2 Leader: 11 Replicas: 11,12,13 Isr: >>>>> 11,12,13 >>>>> > > > >>>>> > > > >>>>> > > > Consumer: >>>>> > > > >>>>> > > > kafka-console-consumer.sh --bootstrap-server n1:9093 >>>>> --consumer.config >>>>> > > > /home/kafka/config/consumer.n1.properties --topic test02 >>>>> > > --from-beginning >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > Producer: >>>>> > > > >>>>> > > > kafka-console-producer.sh --broker-list n1:9093 >>>>> --producer.config >>>>> > > > /homey/kafka/config/producer.n1.properties --sync --topic test02 >>>>> > > > >>>>> > > > ERROR Error when sending message to topic test02 with key: null, >>>>> > value: 0 >>>>> > > > bytes with error: >>>>> > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCal >>>>> lback) >>>>> > > > >>>>> > > > org.apache.kafka.common.errors.TimeoutException: Expiring 1 >>>>> record(s) >>>>> > > for >>>>> > > > test02-1: 1506 ms has passed since batch creation plus linger >>>>> time >>>>> > > > >>>>> > > > >>>>> > > > How to resolve it? >>>>> > > > >>>>> > > > Regards >>>>> > > > >>>>> > > >>>>> > >>>>> >>>> >>>> >>> >> >