FYI, about zookeeper, I used my existing zookeeper (as I have existing zookeeper up and running, which is also used for hbase)
zookeeper versoom: 3.4.10 zoo.cfg ###### tickTime=2000 initLimit=10 syncLimit=5 dataDir=/usr/local/zookeeper/data dataLogDir=/usr/local/zookeeper/datalog clientPort=2181 maxClientCnxns=60 server.1=n1.test.com:2888:3888 server.2=n2.test.com:2888:3888 server.3=n3.test.com:2888:3888 authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider jaasLoginRenew=3600000 requireClientAuthScheme=sasl zookeeper.allowSaslFailedClients=false kerberos.removeHostFromPrincipal=true ###### On Thu, Aug 10, 2017 at 4:35 AM, Ascot Moss <[email protected]> wrote: > server.properties > > ###### > > broker.id=11 > > port=9093 > > host.name=n1 > > advertised.host.name=192.168.0.11 > > allow.everyone.if.no.acl.found=true > > super.users=User:CN=n1.test.com,OU=TEST,O=TEST,L=TEST,ST=TEST,C=TEST > > listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/> > > advertised.listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/> > > ssl.client.auth=required > > ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 > > ssl.keystore.type=JKS > > ssl.truststore.type=JKS > > security.inter.broker.protocol=SSL > > ssl.keystore.location=/home/kafka/kafka.server.keystore.jks > > ssl.keystore.password=Test2017 > > ssl.key.password=Test2017 > > ssl.truststore.location=/home/kafka/kafka.server.truststore.jks > > ssl.truststore.password=Test2017 > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer > > principal.builder.class=org.apache.kafka.common.security. > auth.DefaultPrincipalBuilder > > num.replica.fetchers=4 > > replica.fetch.max.bytes=1048576 > > replica.fetch.wait.max.ms=500 > > replica.high.watermark.checkpoint.interval.ms=5000 > > replica.socket.timeout.ms=30000 > > replica.socket.receive.buffer.bytes=65536 > > replica.lag.time.max.ms=10000 > > controller.socket.timeout.ms=30000 > > controller.message.queue.size=10 > > default.replication.factor=3 > > log.dirs=/usr/log/kafka > > kafka.logs.dir=/usr/log/kafka > > num.partitions=20 > > message.max.bytes=1000000 > > auto.create.topics.enable=true > > log.index.interval.bytes=4096 > > log.index.size.max.bytes=10485760 > > log.retention.hours=720 > > log.flush.interval.ms=10000 > > log.flush.interval.messages=20000 > > log.flush.scheduler.interval.ms=2000 > > log.roll.hours=168 > > log.retention.check.interval.ms=300000 > > log.segment.bytes=1073741824 > > delete.topic.enable=true > > socket.request.max.bytes=104857600 > > socket.receive.buffer.bytes=1048576 > > socket.send.buffer.bytes=1048576 > > num.io.threads=8 > > num.network.threads=8 > > queued.max.requests=16 > > fetch.purgatory.purge.interval.requests=100 > > producer.purgatory.purge.interval.requests=100 > > zookeeper.connect=n1:2181,n2:2181,n3:2181 > > zookeeper.connection.timeout.ms=2000 > > zookeeper.sync.time.ms=2000 > > ###### > > > > > > producer.properties > > ###### > > bootstrap.servers=n1.test.com:9093 <http://n1.test.com:9092/> > > security.protocol=SSL > > ssl.truststore.location=/home/kafka/kafka.client.truststore.jks > > ssl.truststore.password=testkafka > > ssl.keystore.location=/home/kafka/kafka.client.keystore.jks > > ssl.keystore.password=testkafka > > ssl.key.password=testkafka > ##### > > > (I had tried to switch to another port, 9093 is the correct port) > > On Thu, Aug 10, 2017 at 4:28 AM, M. Manna <[email protected]> wrote: > >> Your openssl test is showing connected with port 9092. but your previous >> messages show 9093 - is there some typo issues? Where is SSL running >> >> Please share the following and don't leave any details out. This will only >> create more assumptions. >> >> 1) server.properties >> 2) Zookeeper.properties >> >> Also, run the following command (when the cluster is running) >> zookeeper-shell.sh localhost:2181 >> get /brokers/ids/11 >> >> Does it show that your broker #11 is connected? >> >> >> >> >> On 9 August 2017 at 21:17, Ascot Moss <[email protected]> wrote: >> >> > Dear Manna, >> > >> > >> > What's the status of your SSL? Have you verified that the setup is >> working? >> > Yes, I used " >> > >> > openssl s_client -debug -connect n1.test.com:9092 -tls1 >> > Output: >> > >> > CONNECTED(00000003) >> > >> > write to 0x853e70 [0x89fd43] (155 bytes => 155 (0x9B)) >> > >> > 0000 - 16 03 01 00 96 01 00 00-92 03 01 59 8b 6d 0d b1 >> ...........Y.m.. >> > ... >> > >> > Server certificate >> > >> > -----BEGIN CERTIFICATE----- >> > >> > CwwCSEsxGT............ >> > >> > -----END CERTIFICATE----- >> > >> > --- >> > >> > SSL handshake has read 2470 bytes and written 161 bytes >> > >> > --- >> > >> > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA >> > >> > PSK identity hint: None >> > >> > Start Time: 1502309645 >> > >> > Timeout : 7200 (sec) >> > >> > Verify return code: 19 (self signed certificate in certificate >> chain) >> > >> > --- >> > >> > Regards >> > >> > On Wed, Aug 9, 2017 at 10:29 PM, M. Manna <[email protected]> wrote: >> > >> > > Hi, >> > > >> > > What's the status of your SSL? Have you verified that the setup is >> > working? >> > > >> > > You can enable rough logins using log4j.properties file supplier with >> > kafka >> > > and set the root logging level to DEBUG. This prints out more info to >> > trace >> > > things. Also, you can enable security logging by adding >> > > -Djavax.security.debug=all >> > > >> > > Please share your producer/broker configs with us. >> > > >> > > Kindest Regards, >> > > M. Manna >> > > >> > > On 9 August 2017 at 14:38, Ascot Moss <[email protected]> wrote: >> > > >> > > > Hi, >> > > > >> > > > >> > > > I have setup Kafka 0.10.2.1 with SSL. >> > > > >> > > > >> > > > Check Status: >> > > > >> > > > openssl s_client -debug -connect n1:9093 -tls1 >> > > > >> > > > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA >> > > > >> > > > ... SSL-Session: >> > > > >> > > > Protocol : TLSv1 >> > > > >> > > > PSK identity hint: None >> > > > >> > > > Start Time: 1502285690 >> > > > >> > > > Timeout : 7200 (sec) >> > > > >> > > > Verify return code: 19 (self signed certificate in certificate >> > chain) >> > > > >> > > > >> > > > Create Topic: >> > > > >> > > > kafka-topics.sh --create --zookeeper n1:2181,n2:2181,n3:2181 >> > > > --replication-factor 3 --partitions 3 --topic test02 >> > > > >> > > > ERROR [ReplicaFetcherThread-2-111], Error for partition [test02,2] >> to >> > > > broker 1:org.apache.kafka.common.errors.UnknownTopicOrPartitionExce >> pti >> > > on: >> > > > This server does not host this topic-partition. >> > > > (kafka.server.ReplicaFetcherThread) >> > > > >> > > > However, if I run describe topic, I can see it is created >> > > > >> > > > >> > > > >> > > > Describe Topic: >> > > > >> > > > kafka-topics.sh --zookeeper n1:2181,n2:2181,n3:2181 --describe >> --topic >> > > > test02 >> > > > >> > > > Topic:test02 PartitionCount:3 ReplicationFactor:3 Configs: >> > > > >> > > > Topic: test02 Partition: 0 Leader: 12 Replicas: 12,13,11 Isr: >> 12,13,11 >> > > > >> > > > Topic: test02 Partition: 1 Leader: 13 Replicas: 13,11,12 Isr: >> 13,11,12 >> > > > >> > > > Topic: test02 Partition: 2 Leader: 11 Replicas: 11,12,13 Isr: >> 11,12,13 >> > > > >> > > > >> > > > Consumer: >> > > > >> > > > kafka-console-consumer.sh --bootstrap-server n1:9093 >> --consumer.config >> > > > /home/kafka/config/consumer.n1.properties --topic test02 >> > > --from-beginning >> > > > >> > > > >> > > > >> > > > Producer: >> > > > >> > > > kafka-console-producer.sh --broker-list n1:9093 --producer.config >> > > > /homey/kafka/config/producer.n1.properties --sync --topic test02 >> > > > >> > > > ERROR Error when sending message to topic test02 with key: null, >> > value: 0 >> > > > bytes with error: >> > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) >> > > > >> > > > org.apache.kafka.common.errors.TimeoutException: Expiring 1 >> record(s) >> > > for >> > > > test02-1: 1506 ms has passed since batch creation plus linger time >> > > > >> > > > >> > > > How to resolve it? >> > > > >> > > > Regards >> > > > >> > > >> > >> > >
