( I have 3 test nodes)

get /brokers/ids/11

{"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL://
n1.test.com:9093
"],"jmx_port":-1,"host":null,"timestamp":"1502310695312","port":-1,"version":4}

cZxid = 0x40002787d

ctime = Thu Aug 10 04:31:37 HKT 2017

mZxid = 0x40002787d

mtime = Thu Aug 10 04:31:37 HKT 2017

pZxid = 0x40002787d

cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x35d885c689c00a6

dataLength = 168

numChildren = 0


get /brokers/ids/12

{"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL://
n2.test.com:9093
"],"jmx_port":-1,"host":null,"timestamp":"1502284073115","port":-1,"version":4}

cZxid = 0x400026c66

ctime = Wed Aug 09 21:07:53 HKT 2017

mZxid = 0x400026c66

mtime = Wed Aug 09 21:07:53 HKT 2017

pZxid = 0x400026c66

cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x25d6b41469a0110

dataLength = 168

numChildren = 0


get /brokers/ids/13

{"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL://
n3.test.com:9093
"],"jmx_port":-1,"host":null,"timestamp":"1502284080461","port":-1,"version":4}

cZxid = 0x400026c6c

ctime = Wed Aug 09 21:07:59 HKT 2017

mZxid = 0x400026c6c

mtime = Wed Aug 09 21:07:59 HKT 2017

pZxid = 0x400026c6c

cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x35d885c689c00a2

dataLength = 168

numChildren = 0

On Thu, Aug 10, 2017 at 5:03 AM, Ascot Moss <ascot.m...@gmail.com> wrote:

>
> About:
> zookeeper-shell.sh localhost:2181
> get /brokers/ids/11
>
>
> The result:
>
> zookeeper-shell.sh n1.test.com:2181
>
> Connecting to n1.test.com:2181
>
> Welcome to ZooKeeper!
>
> JLine support is disabled
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
>
> WATCHER::
>
>
>
>
> get /brokers/ids/11
>
> WatchedEvent state:SaslAuthenticated type:None path:null
>
> {"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL://
> n1.test.com:9093"],"jmx_port":-1,"host":null,"timestamp":"1502310695312","
> port":-1,"version":4}
>
> cZxid = 0x40002787d
>
> ctime = Thu Aug 10 04:31:37 HKT 2017
>
> mZxid = 0x40002787d
>
> mtime = Thu Aug 10 04:31:37 HKT 2017
>
> pZxid = 0x40002787d
>
> cversion = 0
>
> dataVersion = 0
>
> aclVersion = 0
>
> ephemeralOwner = 0x35d885c689c00a6
>
> dataLength = 168
>
> numChildren = 0
>
> On Thu, Aug 10, 2017 at 4:46 AM, Ascot Moss <ascot.m...@gmail.com> wrote:
>
>> About:  zookeeper-shell.sh localhost:2181
>> get /brokers/ids/11
>>
>> The result:
>>
>> zookeeper-shell.sh n1.test.com:2181
>>
>> Connecting to n1.test.com:2181
>>
>> Welcome to ZooKeeper!
>>
>> JLine support is disabled
>>
>> WATCHER::
>>
>> WatchedEvent state:SyncConnected type:None path:null
>>
>> WATCHER::
>>
>> WatchedEvent state:SaslAuthenticated type:None path:null
>>
>>
>> On Thu, Aug 10, 2017 at 4:43 AM, Ascot Moss <ascot.m...@gmail.com> wrote:
>>
>>> FYI, about zookeeper, I used my existing zookeeper (as I have existing
>>> zookeeper up and running, which is also used for hbase)
>>>
>>> zookeeper versoom: 3.4.10
>>>
>>> zoo.cfg
>>> ######
>>>
>>> tickTime=2000
>>>
>>> initLimit=10
>>>
>>> syncLimit=5
>>>
>>> dataDir=/usr/local/zookeeper/data
>>>
>>> dataLogDir=/usr/local/zookeeper/datalog
>>>
>>> clientPort=2181
>>>
>>> maxClientCnxns=60
>>>
>>> server.1=n1.test.com:2888:3888
>>>
>>> server.2=n2.test.com:2888:3888
>>>
>>> server.3=n3.test.com:2888:3888
>>>
>>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenti
>>> cationProvider
>>>
>>> jaasLoginRenew=3600000
>>>
>>> requireClientAuthScheme=sasl
>>>
>>> zookeeper.allowSaslFailedClients=false
>>>
>>> kerberos.removeHostFromPrincipal=true
>>>
>>> ######
>>>
>>>
>>>
>>> On Thu, Aug 10, 2017 at 4:35 AM, Ascot Moss <ascot.m...@gmail.com>
>>> wrote:
>>>
>>>> server.properties
>>>>
>>>> ######
>>>>
>>>> broker.id=11
>>>>
>>>> port=9093
>>>>
>>>> host.name=n1
>>>>
>>>> advertised.host.name=192.168.0.11
>>>>
>>>> allow.everyone.if.no.acl.found=true
>>>>
>>>> super.users=User:CN=n1.test.com,OU=TEST,O=TEST,L=TEST,ST=TEST,C=TEST
>>>>
>>>> listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/>
>>>>
>>>> advertised.listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/>
>>>>
>>>> ssl.client.auth=required
>>>>
>>>> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>>
>>>> ssl.keystore.type=JKS
>>>>
>>>> ssl.truststore.type=JKS
>>>>
>>>> security.inter.broker.protocol=SSL
>>>>
>>>> ssl.keystore.location=/home/kafka/kafka.server.keystore.jks
>>>>
>>>> ssl.keystore.password=Test2017
>>>>
>>>> ssl.key.password=Test2017
>>>>
>>>> ssl.truststore.location=/home/kafka/kafka.server.truststore.jks
>>>>
>>>> ssl.truststore.password=Test2017
>>>>
>>>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>>
>>>> principal.builder.class=org.apache.kafka.common.security.aut
>>>> h.DefaultPrincipalBuilder
>>>>
>>>> num.replica.fetchers=4
>>>>
>>>> replica.fetch.max.bytes=1048576
>>>>
>>>> replica.fetch.wait.max.ms=500
>>>>
>>>> replica.high.watermark.checkpoint.interval.ms=5000
>>>>
>>>> replica.socket.timeout.ms=30000
>>>>
>>>> replica.socket.receive.buffer.bytes=65536
>>>>
>>>> replica.lag.time.max.ms=10000
>>>>
>>>> controller.socket.timeout.ms=30000
>>>>
>>>> controller.message.queue.size=10
>>>>
>>>> default.replication.factor=3
>>>>
>>>> log.dirs=/usr/log/kafka
>>>>
>>>> kafka.logs.dir=/usr/log/kafka
>>>>
>>>> num.partitions=20
>>>>
>>>> message.max.bytes=1000000
>>>>
>>>> auto.create.topics.enable=true
>>>>
>>>> log.index.interval.bytes=4096
>>>>
>>>> log.index.size.max.bytes=10485760
>>>>
>>>> log.retention.hours=720
>>>>
>>>> log.flush.interval.ms=10000
>>>>
>>>> log.flush.interval.messages=20000
>>>>
>>>> log.flush.scheduler.interval.ms=2000
>>>>
>>>> log.roll.hours=168
>>>>
>>>> log.retention.check.interval.ms=300000
>>>>
>>>> log.segment.bytes=1073741824
>>>>
>>>> delete.topic.enable=true
>>>>
>>>> socket.request.max.bytes=104857600
>>>>
>>>> socket.receive.buffer.bytes=1048576
>>>>
>>>> socket.send.buffer.bytes=1048576
>>>>
>>>> num.io.threads=8
>>>>
>>>> num.network.threads=8
>>>>
>>>> queued.max.requests=16
>>>>
>>>> fetch.purgatory.purge.interval.requests=100
>>>>
>>>> producer.purgatory.purge.interval.requests=100
>>>>
>>>> zookeeper.connect=n1:2181,n2:2181,n3:2181
>>>>
>>>> zookeeper.connection.timeout.ms=2000
>>>>
>>>> zookeeper.sync.time.ms=2000
>>>>
>>>> ######
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> producer.properties
>>>>
>>>> ######
>>>>
>>>> bootstrap.servers=n1.test.com:9093 <http://n1.test.com:9092/>
>>>>
>>>> security.protocol=SSL
>>>>
>>>> ssl.truststore.location=/home/kafka/kafka.client.truststore.jks
>>>>
>>>> ssl.truststore.password=testkafka
>>>>
>>>> ssl.keystore.location=/home/kafka/kafka.client.keystore.jks
>>>>
>>>> ssl.keystore.password=testkafka
>>>>
>>>> ssl.key.password=testkafka
>>>> #####
>>>>
>>>>
>>>> (I had tried to switch to another port, 9093 is the correct port)
>>>>
>>>> On Thu, Aug 10, 2017 at 4:28 AM, M. Manna <manme...@gmail.com> wrote:
>>>>
>>>>> Your openssl test is showing connected with port 9092. but your
>>>>> previous
>>>>> messages show 9093 - is there some typo issues? Where is SSL running
>>>>>
>>>>> Please share the following and don't leave any details out. This will
>>>>> only
>>>>> create more assumptions.
>>>>>
>>>>> 1) server.properties
>>>>> 2) Zookeeper.properties
>>>>>
>>>>> Also, run the following command (when the cluster is running)
>>>>> zookeeper-shell.sh localhost:2181
>>>>> get /brokers/ids/11
>>>>>
>>>>> Does it show that your broker #11 is connected?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 9 August 2017 at 21:17, Ascot Moss <ascot.m...@gmail.com> wrote:
>>>>>
>>>>> > Dear Manna,
>>>>> >
>>>>> >
>>>>> > What's the status of your SSL? Have you verified that the setup is
>>>>> working?
>>>>> > Yes, I used "
>>>>> >
>>>>> > openssl s_client -debug -connect n1.test.com:9092 -tls1
>>>>> > Output:
>>>>> >
>>>>> > CONNECTED(00000003)
>>>>> >
>>>>> > write to 0x853e70 [0x89fd43] (155 bytes => 155 (0x9B))
>>>>> >
>>>>> > 0000 - 16 03 01 00 96 01 00 00-92 03 01 59 8b 6d 0d b1
>>>>>  ...........Y.m..
>>>>> > ...
>>>>> >
>>>>> > Server certificate
>>>>> >
>>>>> > -----BEGIN CERTIFICATE-----
>>>>> >
>>>>> > CwwCSEsxGT............
>>>>> >
>>>>> > -----END CERTIFICATE-----
>>>>> >
>>>>> > ---
>>>>> >
>>>>> > SSL handshake has read 2470 bytes and written 161 bytes
>>>>> >
>>>>> > ---
>>>>> >
>>>>> > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
>>>>> >
>>>>> >     PSK identity hint: None
>>>>> >
>>>>> >     Start Time: 1502309645
>>>>> >
>>>>> >     Timeout   : 7200 (sec)
>>>>> >
>>>>> >     Verify return code: 19 (self signed certificate in certificate
>>>>> chain)
>>>>> >
>>>>> > ---
>>>>> >
>>>>> > Regards
>>>>> >
>>>>> > On Wed, Aug 9, 2017 at 10:29 PM, M. Manna <manme...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > Hi,
>>>>> > >
>>>>> > > What's the status of your SSL? Have you verified that the setup is
>>>>> > working?
>>>>> > >
>>>>> > > You can enable rough logins using log4j.properties file supplier
>>>>> with
>>>>> > kafka
>>>>> > > and set the root logging level to DEBUG. This prints out more info
>>>>> to
>>>>> > trace
>>>>> > > things. Also, you can enable security logging by adding
>>>>> > > -Djavax.security.debug=all
>>>>> > >
>>>>> > > Please share your producer/broker configs with us.
>>>>> > >
>>>>> > > Kindest Regards,
>>>>> > > M. Manna
>>>>> > >
>>>>> > > On 9 August 2017 at 14:38, Ascot Moss <ascot.m...@gmail.com>
>>>>> wrote:
>>>>> > >
>>>>> > > > Hi,
>>>>> > > >
>>>>> > > >
>>>>> > > > I have setup Kafka 0.10.2.1 with SSL.
>>>>> > > >
>>>>> > > >
>>>>> > > > Check Status:
>>>>> > > >
>>>>> > > > openssl s_client -debug -connect n1:9093 -tls1
>>>>> > > >
>>>>> > > > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
>>>>> > > >
>>>>> > > > ... SSL-Session:
>>>>> > > >
>>>>> > > >     Protocol  : TLSv1
>>>>> > > >
>>>>> > > >     PSK identity hint: None
>>>>> > > >
>>>>> > > >     Start Time: 1502285690
>>>>> > > >
>>>>> > > >     Timeout   : 7200 (sec)
>>>>> > > >
>>>>> > > >     Verify return code: 19 (self signed certificate in
>>>>> certificate
>>>>> > chain)
>>>>> > > >
>>>>> > > >
>>>>> > > > Create Topic:
>>>>> > > >
>>>>> > > > kafka-topics.sh --create --zookeeper n1:2181,n2:2181,n3:2181
>>>>> > > > --replication-factor 3 --partitions 3 --topic test02
>>>>> > > >
>>>>> > > > ERROR [ReplicaFetcherThread-2-111], Error for partition
>>>>> [test02,2] to
>>>>> > > > broker 1:org.apache.kafka.common.erro
>>>>> rs.UnknownTopicOrPartitionExcepti
>>>>> > > on:
>>>>> > > > This server does not host this topic-partition.
>>>>> > > > (kafka.server.ReplicaFetcherThread)
>>>>> > > >
>>>>> > > > However, if I run describe topic, I can see it is created
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > Describe Topic:
>>>>> > > >
>>>>> > > > kafka-topics.sh --zookeeper n1:2181,n2:2181,n3:2181 --describe
>>>>> --topic
>>>>> > > > test02
>>>>> > > >
>>>>> > > > Topic:test02 PartitionCount:3 ReplicationFactor:3 Configs:
>>>>> > > >
>>>>> > > > Topic: test02 Partition: 0 Leader: 12 Replicas: 12,13,11 Isr:
>>>>> 12,13,11
>>>>> > > >
>>>>> > > > Topic: test02 Partition: 1 Leader: 13 Replicas: 13,11,12 Isr:
>>>>> 13,11,12
>>>>> > > >
>>>>> > > > Topic: test02 Partition: 2 Leader: 11 Replicas: 11,12,13 Isr:
>>>>> 11,12,13
>>>>> > > >
>>>>> > > >
>>>>> > > > Consumer:
>>>>> > > >
>>>>> > > > kafka-console-consumer.sh --bootstrap-server n1:9093
>>>>> --consumer.config
>>>>> > > > /home/kafka/config/consumer.n1.properties --topic test02
>>>>> > > --from-beginning
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > Producer:
>>>>> > > >
>>>>> > > > kafka-console-producer.sh --broker-list n1:9093
>>>>> --producer.config
>>>>> > > > /homey/kafka/config/producer.n1.properties --sync --topic test02
>>>>> > > >
>>>>> > > > ERROR Error when sending message to topic test02 with key: null,
>>>>> > value: 0
>>>>> > > > bytes with error:
>>>>> > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCal
>>>>> lback)
>>>>> > > >
>>>>> > > > org.apache.kafka.common.errors.TimeoutException: Expiring 1
>>>>> record(s)
>>>>> > > for
>>>>> > > > test02-1: 1506 ms has passed since batch creation plus linger
>>>>> time
>>>>> > > >
>>>>> > > >
>>>>> > > > How to resolve it?
>>>>> > > >
>>>>> > > > Regards
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Reply via email to