FYI, about zookeeper, I used my existing zookeeper (as I have existing
zookeeper up and running, which is also used for hbase)

zookeeper versoom: 3.4.10

zoo.cfg
######

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/usr/local/zookeeper/data

dataLogDir=/usr/local/zookeeper/datalog

clientPort=2181

maxClientCnxns=60

server.1=n1.test.com:2888:3888

server.2=n2.test.com:2888:3888

server.3=n3.test.com:2888:3888

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

jaasLoginRenew=3600000

requireClientAuthScheme=sasl

zookeeper.allowSaslFailedClients=false

kerberos.removeHostFromPrincipal=true

######



On Thu, Aug 10, 2017 at 4:35 AM, Ascot Moss <ascot.m...@gmail.com> wrote:

> server.properties
>
> ######
>
> broker.id=11
>
> port=9093
>
> host.name=n1
>
> advertised.host.name=192.168.0.11
>
> allow.everyone.if.no.acl.found=true
>
> super.users=User:CN=n1.test.com,OU=TEST,O=TEST,L=TEST,ST=TEST,C=TEST
>
> listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/>
>
> advertised.listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/>
>
> ssl.client.auth=required
>
> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>
> ssl.keystore.type=JKS
>
> ssl.truststore.type=JKS
>
> security.inter.broker.protocol=SSL
>
> ssl.keystore.location=/home/kafka/kafka.server.keystore.jks
>
> ssl.keystore.password=Test2017
>
> ssl.key.password=Test2017
>
> ssl.truststore.location=/home/kafka/kafka.server.truststore.jks
>
> ssl.truststore.password=Test2017
>
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>
> principal.builder.class=org.apache.kafka.common.security.
> auth.DefaultPrincipalBuilder
>
> num.replica.fetchers=4
>
> replica.fetch.max.bytes=1048576
>
> replica.fetch.wait.max.ms=500
>
> replica.high.watermark.checkpoint.interval.ms=5000
>
> replica.socket.timeout.ms=30000
>
> replica.socket.receive.buffer.bytes=65536
>
> replica.lag.time.max.ms=10000
>
> controller.socket.timeout.ms=30000
>
> controller.message.queue.size=10
>
> default.replication.factor=3
>
> log.dirs=/usr/log/kafka
>
> kafka.logs.dir=/usr/log/kafka
>
> num.partitions=20
>
> message.max.bytes=1000000
>
> auto.create.topics.enable=true
>
> log.index.interval.bytes=4096
>
> log.index.size.max.bytes=10485760
>
> log.retention.hours=720
>
> log.flush.interval.ms=10000
>
> log.flush.interval.messages=20000
>
> log.flush.scheduler.interval.ms=2000
>
> log.roll.hours=168
>
> log.retention.check.interval.ms=300000
>
> log.segment.bytes=1073741824
>
> delete.topic.enable=true
>
> socket.request.max.bytes=104857600
>
> socket.receive.buffer.bytes=1048576
>
> socket.send.buffer.bytes=1048576
>
> num.io.threads=8
>
> num.network.threads=8
>
> queued.max.requests=16
>
> fetch.purgatory.purge.interval.requests=100
>
> producer.purgatory.purge.interval.requests=100
>
> zookeeper.connect=n1:2181,n2:2181,n3:2181
>
> zookeeper.connection.timeout.ms=2000
>
> zookeeper.sync.time.ms=2000
>
> ######
>
>
>
>
>
> producer.properties
>
> ######
>
> bootstrap.servers=n1.test.com:9093 <http://n1.test.com:9092/>
>
> security.protocol=SSL
>
> ssl.truststore.location=/home/kafka/kafka.client.truststore.jks
>
> ssl.truststore.password=testkafka
>
> ssl.keystore.location=/home/kafka/kafka.client.keystore.jks
>
> ssl.keystore.password=testkafka
>
> ssl.key.password=testkafka
> #####
>
>
> (I had tried to switch to another port, 9093 is the correct port)
>
> On Thu, Aug 10, 2017 at 4:28 AM, M. Manna <manme...@gmail.com> wrote:
>
>> Your openssl test is showing connected with port 9092. but your previous
>> messages show 9093 - is there some typo issues? Where is SSL running
>>
>> Please share the following and don't leave any details out. This will only
>> create more assumptions.
>>
>> 1) server.properties
>> 2) Zookeeper.properties
>>
>> Also, run the following command (when the cluster is running)
>> zookeeper-shell.sh localhost:2181
>> get /brokers/ids/11
>>
>> Does it show that your broker #11 is connected?
>>
>>
>>
>>
>> On 9 August 2017 at 21:17, Ascot Moss <ascot.m...@gmail.com> wrote:
>>
>> > Dear Manna,
>> >
>> >
>> > What's the status of your SSL? Have you verified that the setup is
>> working?
>> > Yes, I used "
>> >
>> > openssl s_client -debug -connect n1.test.com:9092 -tls1
>> > Output:
>> >
>> > CONNECTED(00000003)
>> >
>> > write to 0x853e70 [0x89fd43] (155 bytes => 155 (0x9B))
>> >
>> > 0000 - 16 03 01 00 96 01 00 00-92 03 01 59 8b 6d 0d b1
>>  ...........Y.m..
>> > ...
>> >
>> > Server certificate
>> >
>> > -----BEGIN CERTIFICATE-----
>> >
>> > CwwCSEsxGT............
>> >
>> > -----END CERTIFICATE-----
>> >
>> > ---
>> >
>> > SSL handshake has read 2470 bytes and written 161 bytes
>> >
>> > ---
>> >
>> > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
>> >
>> >     PSK identity hint: None
>> >
>> >     Start Time: 1502309645
>> >
>> >     Timeout   : 7200 (sec)
>> >
>> >     Verify return code: 19 (self signed certificate in certificate
>> chain)
>> >
>> > ---
>> >
>> > Regards
>> >
>> > On Wed, Aug 9, 2017 at 10:29 PM, M. Manna <manme...@gmail.com> wrote:
>> >
>> > > Hi,
>> > >
>> > > What's the status of your SSL? Have you verified that the setup is
>> > working?
>> > >
>> > > You can enable rough logins using log4j.properties file supplier with
>> > kafka
>> > > and set the root logging level to DEBUG. This prints out more info to
>> > trace
>> > > things. Also, you can enable security logging by adding
>> > > -Djavax.security.debug=all
>> > >
>> > > Please share your producer/broker configs with us.
>> > >
>> > > Kindest Regards,
>> > > M. Manna
>> > >
>> > > On 9 August 2017 at 14:38, Ascot Moss <ascot.m...@gmail.com> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > >
>> > > > I have setup Kafka 0.10.2.1 with SSL.
>> > > >
>> > > >
>> > > > Check Status:
>> > > >
>> > > > openssl s_client -debug -connect n1:9093 -tls1
>> > > >
>> > > > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
>> > > >
>> > > > ... SSL-Session:
>> > > >
>> > > >     Protocol  : TLSv1
>> > > >
>> > > >     PSK identity hint: None
>> > > >
>> > > >     Start Time: 1502285690
>> > > >
>> > > >     Timeout   : 7200 (sec)
>> > > >
>> > > >     Verify return code: 19 (self signed certificate in certificate
>> > chain)
>> > > >
>> > > >
>> > > > Create Topic:
>> > > >
>> > > > kafka-topics.sh --create --zookeeper n1:2181,n2:2181,n3:2181
>> > > > --replication-factor 3 --partitions 3 --topic test02
>> > > >
>> > > > ERROR [ReplicaFetcherThread-2-111], Error for partition [test02,2]
>> to
>> > > > broker 1:org.apache.kafka.common.errors.UnknownTopicOrPartitionExce
>> pti
>> > > on:
>> > > > This server does not host this topic-partition.
>> > > > (kafka.server.ReplicaFetcherThread)
>> > > >
>> > > > However, if I run describe topic, I can see it is created
>> > > >
>> > > >
>> > > >
>> > > > Describe Topic:
>> > > >
>> > > > kafka-topics.sh --zookeeper n1:2181,n2:2181,n3:2181 --describe
>> --topic
>> > > > test02
>> > > >
>> > > > Topic:test02 PartitionCount:3 ReplicationFactor:3 Configs:
>> > > >
>> > > > Topic: test02 Partition: 0 Leader: 12 Replicas: 12,13,11 Isr:
>> 12,13,11
>> > > >
>> > > > Topic: test02 Partition: 1 Leader: 13 Replicas: 13,11,12 Isr:
>> 13,11,12
>> > > >
>> > > > Topic: test02 Partition: 2 Leader: 11 Replicas: 11,12,13 Isr:
>> 11,12,13
>> > > >
>> > > >
>> > > > Consumer:
>> > > >
>> > > > kafka-console-consumer.sh --bootstrap-server n1:9093
>> --consumer.config
>> > > > /home/kafka/config/consumer.n1.properties --topic test02
>> > > --from-beginning
>> > > >
>> > > >
>> > > >
>> > > > Producer:
>> > > >
>> > > > kafka-console-producer.sh --broker-list n1:9093  --producer.config
>> > > > /homey/kafka/config/producer.n1.properties --sync --topic test02
>> > > >
>> > > > ERROR Error when sending message to topic test02 with key: null,
>> > value: 0
>> > > > bytes with error:
>> > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>> > > >
>> > > > org.apache.kafka.common.errors.TimeoutException: Expiring 1
>> record(s)
>> > > for
>> > > > test02-1: 1506 ms has passed since batch creation plus linger time
>> > > >
>> > > >
>> > > > How to resolve it?
>> > > >
>> > > > Regards
>> > > >
>> > >
>> >
>>
>
>

Reply via email to