Kafka SSL Error

ssl.trustStore.locations=/home/kafka/kafka/ssl/kafka.truststore.jks ssl.trustStore.password=password ssl.keyStore.location=/home/kafka/kafka/ssl/kafka.keystore.jks ssl.keyStore.password=password ssl.key.password=password security.inter.broker.protocol=SASL_SSL ssl.client.auth=required sasl.enabled.mechanisms=SCRAM

Python client failed to connect secured Kafka: SSL handshake failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Hi Kafka Team Recently I moved Kafka cluster from CentOS8 to UbuntuServer20.04, same Kafka version(2.13-3.0.0), same Kafka configuration(check below), same JDK(openjdk-11-jdk) in server, but I get python client failed to connect. # SASL-SSL security.inter.broker.protocol=SASL_SSL

Apache kafka SSL Security

Hi , I have been trying to enable Kafka security ssl authentication using certificates and encryption. but i am getting errors when i try to create a topic and Kafka status fails whereas zookeeper is running fine. Note :- The screenshot of the error I get when I try to create a topic is attached

Kafka SSL CA Change Issue

Hi Team, we need some help regarding ca certificate authority change in kafka . Currently we are connecting in kafka using ssl implementation. kafka version used is 1.1.1 below is server.properties listeners=INT://$PVT_HOST_NAME:9094,EXT://$PVT_HOST_NAME:9092

Error on kafka ssl configurtion

Hi , I have been trying to enable Kafka security ssl authentication using certificates and encryption. but i am getting errors when i try to create a topic and Kafka status fails whereas zookeeper is running fine. Note :- The screenshot of the error I get when I try to create a topic is attached

Re: Kafka SSL

t;> I setup Kafka and client SSL config by taking reference of >> Apache Kafka<https://kafka.apache.org/documentation/#security_ssl> >> Apache Kafka TLS encryption & authentication - Azure HDInsight | >> Microsoft Docs< >> https://docs.microsoft.com/en-us/azure/hd

Re: Kafka SSL

ailed). > > > I setup Kafka and client SSL config by taking reference of > Apache Kafka<https://kafka.apache.org/documentation/#security_ssl> > Apache Kafka TLS encryption & authentication - Azure HDInsight | Microsoft > Docs< > https://docs.microsoft.com/en-us/

Kafka SSL

tps://kafka.apache.org/documentation/#security_ssl> Apache Kafka TLS encryption & authentication - Azure HDInsight | Microsoft Docs<https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-ssl-encryption-authentication> And I can verify my Kafka cluster SSL with below command: openssl

Re: Kafka SSL opening too many connections from client to Broker

ss=com.org.KafkaJsonSerializer > > kafka.producer.topic.audit=Audit > > kafka.producer.topic.audit.test=audit-trail-test > > kafka.producer.topic.crl=certificate-revocation > > kafka.test.to.test.topic.t=Aer > > kafka.producer.topic.data=compacted > > kafka

Kafka SSL opening too many connections from client to Broker

/test/ssl/keystore/kafka.keystore.jks ssl.truststore.location=/ test.com/data/kafka/ssl/truststore/kafka.truststore.jks ssl.key.password=** ssl.keystore.password=** ssl.truststore.password=** security.protocol=SSL ssl.protocol=TLS ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 ssl.keystore.type

Kafka SSL Issue Observed

Hi team, Any update on the below issue. Regards, Soumya From: Nayak, Soumya R. Sent: Wednesday, July 31, 2019 11:37 AM To: users@kafka.apache.org Subject: Kafka SSL Issue Observed Hi team, I am using the SSL and SASL PLAIN on the kafka brokers (cluster of 4 nodes). The version of kafka

Kafka SSL Issue Observed

Hi team, I am using the SSL and SASL PLAIN on the kafka brokers (cluster of 4 nodes). The version of kafka - 1.0.0 . I am observing the below issue with regards to SSL. Why this issue is happening? Is this issue addressed in the latest versions ? [2019-07-30 06:11:35,629] WARN Failed to send

Re: kafka ssl config

If you can access the remote file via a mounted filesystem, you can specify '/mountpoint/truststore.jks’ as the value for ssl.truststore.location. You cannot use a url to specify a remote resource. > On May 2, 2019, at 11:38 AM, anurag wrote: > > Hi All, > > Is it possible to set the value

kafka ssl config

Hi All, Is it possible to set the value of ssl.truststore.location to a location on remote host. Basically I have ssl certificates available on remote host and i would like my docker kafka container to read and use certificates from remote location. If this is possible can you please provide an

Kafka SSL setup - producer timing out

I am trying to set up a three node Kafka v1.1 cluster with SSL. I can consume messages via the SSL port but I cannot produce. The command is timing out after 60 seconds with the following error message: ERROR Error when sending message to topic myTopic with key: null, value: 5 bytes with

Re: Kafka SSL and multiple domain names

t: Thursday, February 14, 2019 5:41 AM To: users@kafka.apache.org Subject: Kafka SSL and multiple domain names Hello, We need to have the same Kafka cluster bound to multiple DNS aliases/domain names. However, for some poor reason, we can't have a single SSL certificate with subject alt names

Kafka SSL and multiple domain names

Hello, We need to have the same Kafka cluster bound to multiple DNS aliases/domain names. However, for some poor reason, we can't have a single SSL certificate with subject alt names matching all DNS aliases. Is it possible to use different SSL certs depending on the hostname used by the client?

Re: Kafka SSL error

: sham singh <singh.shammi2...@gmail.com> Sent: Thursday, December 21, 2017 4:06 PM To: users@kafka.apache.org Subject: Re: Kafka SSL error Ted - i'm not seeing any difference in t

Re: Kafka SSL error

hello, here is the update on this .. seems the script -> */usr/hdp/2.5.3.0-37/kafka/bin/kafka-producer-perf-test.sh * has an issue in SSL mode, it seems to not be able to recognize the security-protocol=SSL & the config file passed i.e. when the truststore, password is passed through the config

Re: Kafka SSL error

Ted - i'm not seeing any difference in the Non-working & working clusters .. Another thing, seem like there is some issue with the connectivity .. the console consumer gets disconnected /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-consumer.sh --new-consumer --topic mmtest1 --bootstrap-server

Re: Kafka SSL error

Since you're using a Vendor's distro, can you post on their community page ? BTW do you notice any difference in settings between the working cluster and this cluster ? Cheers On Thu, Dec 21, 2017 at 12:27 PM, sham singh wrote: > Hello All - > I'm getting this

Kafka SSL error

Hello All - I'm getting this error, when publishing messages to Kafka topic using SSL mode, Command to publish messages : */usr/hdp/2.5.3.0-37/kafka/bin/kafka-producer-perf-test.sh --messages 100 --message-size 1000 --topics mmtest4 \* *--broker-list :9093,:9093,:9093, \* *--threads 1

Re: Kafka SSL error

Thanks Jakub .. for your inputs & help in this ! I was able to get this to work last week.. On Thu, Sep 21, 2017 at 12:22 AM, Jakub Scholz wrote: > Hi, > > If you want the Kafka broker to present the whole chain you have to use the > chain when creating the PKCS12 file (use the

Re: Kafka SSL error

Hi Jakub, Thanks for the detailed note... here is the update -> I was able to convert the host.cert.pem to PKCS#12 & import the cert into the kafka.server.keystore.jks (also into kafka.server.truststore.jks) wrt the host.root.pem & host.intermed.pem certs - i'm assuming i need to convert them

Re: Kafka SSL error

ad 1) The problem is that the signed certificate (host.cert.pem) which the CA provides is only the public key. You have to combine it with the private key which you created when requesting the signed certificate. The private key is never sent to the CA so they cannot provide it back. You or

Re: Kafka SSL error

Hello - thanks for the response Here is the update on the issue. I'm using certs signed/provided by org-wide CA (geotrust, not a self-signed cert) The Signed(by the CA - geotrust) cert provided has 3 certificates - host.chain.pem (certificate chain - contains the Root, Intermediate, Signed Server

Re: Kafka SSL error

Hi, Looking at your commands it looks as if you generated a self signed key for server, self signed key for client and then imported the CA keys public keys into the truststores. I don’t think this will work because now you have two different self signed keys in the keystores and the presumably

Kafka SSL error

Hello All - I was able to set up SSL for the Kafka brokers, using OpenSSL. however, I'm having issues with setting up SSL using the pem file (i.e. SSL certificate - certified by CA, provided by the company) Here is what i've done - created the server/client keystore & truststore files and

Apache Kafka SSL Deployment

Hi All , How can I avoid using password for keystore creation ? We are currently passing keystore password while accessing TLS enabled Kafka instance . I would like to use either passwordless keystore or avoid password for clients accessing Kafka .

Re: Kafka SSL encryption plus external CA

From: Stephane Maarek <steph...@simplemachines.com.au> Sent: Tuesday, December 20, 2016 7:11 PM To: Rajini Sivaram Cc: users@kafka.apache.org Subject: Re: Kafka SSL encryption plus external CA Thanks Rajini. I used a CNAME broker-bootstrap-A.example.com that rou

Re: Kafka SSL encryption plus external CA

Stephane, I believe that should work, though I haven't tried it myself. On Wed, Dec 21, 2016 at 12:11 AM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Thanks Rajini. > > I used a CNAME broker-bootstrap-A.example.com that round robins to the > actual brokers broker-1.example.com,

Re: Kafka SSL encryption plus external CA

Thanks Rajini. I used a CNAME broker-bootstrap-A.example.com that round robins to the actual brokers broker-1.example.com, broker-2.example.com (etc etc). Therefore no brokers advertises the bootstrap DNS name we’re using. Is that an issue? The SSL certificate wildcard will match both boostrap

Re: Kafka SSL encryption plus external CA

Stephane, Bootstrap brokers are also verified by the client in exactly the same way, so they should also match the wildcard of their certificate. Basically, clients need to make a secure SSL connection to one of the bootstrap brokers to obtain advertised hostnames of brokers, so they need to

Re: Kafka SSL encryption plus external CA

Thanks Rajini! Also, I currently have each broker advertising as broker1.mydomain.com, broker2.mydomain.com broker6.mydomain.com etc… I have setup CNAME with round robin fashion to group brokers by availability zone i.e. broker-a.mydomain.com broker-b.mydomain.com broker-c.mydomain.com. I use

Re: Kafka SSL encryption plus external CA

Stephane, If you are using a trusted CA like Verisign, clients don't need to specify a truststore. The host names specified in advertised.listeners in the broker must match the wildcard DNS names in the certificates if clients configure ssl.endpoint.identification.algorithm=https. If

Kafka SSL encryption plus external CA

Hi, I have read the docs extensively but yet there are a few answers I can’t find. It has to do with external CA Please confirm my understanding if possible: I can create my own CA to sign all the brokers and clients certificates. Pros: - cheap, easy, automated. I need to find a way to access

Re: Can Kafka/SSL be terminated at a load balancer?

PlainLoginModule required >> >> username="someuser" >> >> user_kafka="somePassword" >> >> password="kafka-password"; >> >> }; >> >> >> The fact that I can no longer even consume from a topic over P

Re: Can Kafka/SSL be terminated at a load balancer?

t; (which is a regression of where I was before we started trying to add SSL) > tells me there is something wrong in either server.properties or jaas.conf. > I've checked the Kafka broker logs (server.log) each time I try connecting > and this is the only line that gets printed: > > >

Re: Can Kafka/SSL be terminated at a load balancer?

rajinisiva...@googlemail.com> Sent: Monday, November 21, 2016 11:03:14 AM To: users@kafka.apache.org Subject: Re: Can Kafka/SSL be terminated at a load balancer? Rule #1 and Rule #2 cannot co-exist. You are basically configuring your LB to point to a Kafka broker and you are pointing eac

Re: Can Kafka/SSL be terminated at a load balancer?

I was really asking was: does that exception > (ClosedChannelException) indicate bad configs on the Kafka broker? > > > From: Zac Harvey <zac.har...@welltok.com> > Sent: Thursday, November 17, 2016 4:44:06 PM > To: users@kafka.apache.org > Subjec

Re: Can Kafka/SSL be terminated at a load balancer?

configs on the Kafka broker? From: Zac Harvey <zac.har...@welltok.com> Sent: Thursday, November 17, 2016 4:44:06 PM To: users@kafka.apache.org Subject: Can Kafka/SSL be terminated at a load balancer? We have two Kafka nodes and for reasons outside of this qu

Re: Can Kafka/SSL be terminated at a load balancer?

ubleshoot it? Thanks again! Best, Zac From: Rajini Sivaram <rajinisiva...@googlemail.com> Sent: Monday, November 21, 2016 10:11:00 AM To: users@kafka.apache.org Subject: Re: Can Kafka/SSL be terminated at a load balancer? A load balancer that balances the load a

Re: Can Kafka/SSL be terminated at a load balancer?

<mgai...@hotmail.com> > wrote: > > > > > > > > > > > > > From: Zac Harvey <zac.har...@welltok.com> > > Sent: Monday, November 21, 2016 8:59 AM > > To: users@kafka.apache.org > > Subject: Re: C

Re: Can Kafka/SSL be terminated at a load balancer?

ntext/9093? > > > Thanks again, just still a little uncertain about the traffic/ports coming > into the load balancer! > > > Best, > > Zac > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > Sent: Monday, November 21,

Re: Can Kafka/SSL be terminated at a load balancer?

<mgai...@hotmail.com> wrote: > > > > > > From: Zac Harvey <zac.har...@welltok.com> > Sent: Monday, November 21, 2016 8:59 AM > To: users@kafka.apache.org > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > T

Re: Can Kafka/SSL be terminated at a load balancer?

From: Zac Harvey <zac.har...@welltok.com> Sent: Monday, November 21, 2016 8:59 AM To: users@kafka.apache.org Subject: Re: Can Kafka/SSL be terminated at a load balancer? Thanks again Rajini, Using these configs, would clients connect to the load ba

Re: Can Kafka/SSL be terminated at a load balancer?

balancer! Best, Zac From: Rajini Sivaram <rajinisiva...@googlemail.com> Sent: Monday, November 21, 2016 8:48:41 AM To: users@kafka.apache.org Subject: Re: Can Kafka/SSL be terminated at a load balancer? Zac, Yes, that is correct. Ruby c

Re: Can Kafka/SSL be terminated at a load balancer?

ed to authenticate, correct? > > > Thanks again for all the great help so far, you've already helped me more > than you know! > > > Zac > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > Sent: Monday, November 21, 2016 3:

Re: Can Kafka/SSL be terminated at a load balancer?

text://:9092 > advertised.listeners=plaintext://mykafka01.example.com:9092 > > Thanks again! > Zac > > > > > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > Sent: Friday, November 18, 2016 9:57:22 AM > To: users@kafka.apa

Re: Can Kafka/SSL be terminated at a load balancer?

ykafka01.example.com:9092 > > Thanks again! > Zac > > > > > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > Sent: Friday, November 18, 2016 9:57:22 AM > To: users@kafka.apache.org > Subject: Re: Can Kafka/SSL be

Re: Can Kafka/SSL be terminated at a load balancer?

<rajinisiva...@googlemail.com> Sent: Friday, November 18, 2016 9:57:22 AM To: users@kafka.apache.org Subject: Re: Can Kafka/SSL be terminated at a load balancer? You should set advertised.listeners rather than the older advertised.host.name property in server.properties: - lis

Re: Can Kafka/SSL be terminated at a load balancer?

figs that will need to be made for the Ruby > clients to connect over SSL? > > > Thank you enormously here! > > > Best, > > Zac > > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > Sent: Friday, November 18,

Re: Can Kafka/SSL be terminated at a load balancer?

, Zac From: Rajini Sivaram <rajinisiva...@googlemail.com> Sent: Friday, November 18, 2016 5:15:13 AM To: users@kafka.apache.org Subject: Re: Can Kafka/SSL be terminated at a load balancer? Zac, Kafka has its own built-in load-balancing mechanism based on partition assignment. Requests a

Re: Can Kafka/SSL be terminated at a load balancer?

Zac, Kafka has its own built-in load-balancing mechanism based on partition assignment. Requests are processed by partition leaders, distributing load across the brokers in the cluster. If you want to put a proxy like HAProxy with SSL termination in front of your brokers for added security, you

Can Kafka/SSL be terminated at a load balancer?

We have two Kafka nodes and for reasons outside of this question, would like to set up a load balancer to terminate SSL with producers (clients). The SSL cert hosted by the load balancer will be signed by trusted/root CA that clients should natively trust. Is this possible to do, or does

Re: Kafka SSL Configuration Problems

Aha , got it. So thats where I got confused. > On Feb 1, 2016, at 3:04 PM, Ismael Juma wrote: > > Hi Nazario, > > The problem in the original post is that you were setting > advertised.host.name, which means that advertised.listeners won't fall back > to listeners

Re: Kafka SSL Configuration Problems

So I made the port 9092 but SSL. But it seems like it is just openning it for PLAINTEXT. Even though it has registered it as SSL [2016-02-01 13:42:20,536] INFO Registered broker 0 at path /brokers/ids/0 with addresses: SSL -> EndPoint(reactor.us.cixsoft.net,9092,SSL) (kafka.utils.ZkUtils)

Re: Kafka SSL Configuration Problems

I dont think that is the behavior I have seen. If I set listeners only ( as per my original post) , SSL will never get registered. [2016-02-01 11:27:49,712] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT) (kafka.utils.ZkUtils)

Re: Kafka SSL Configuration Problems

So it looks like you need both listeners and advertised.listeners ..? When I set both configs .. It finally worked. Maybe we can update the docs ..? > On Feb 1, 2016, at 1:59 PM, Nazario Parsacala wrote: > > So I made the port 9092 but SSL. But it seems like it is

Re: Kafka SSL Configuration Problems

On Mon, Feb 1, 2016 at 7:15 PM, Nazario Parsacala wrote: > So it looks like you need both listeners and advertised.listeners ..? > No, you always need to set `listeners` (`advertised.listeners` defaults to `listeners`). If you want `advertised.listeners` to be different

Re: Kafka SSL Configuration Problems

Hi Nazario, The problem in the original post is that you were setting advertised.host.name, which means that advertised.listeners won't fall back to listeners anymore. Yes, it's bit confusing given how the configs evolved over time. I have configured several clusters to use SSL by setting

Kafka SSL Configuration Problems

Hi, We were using kafka for a while now. We have been using the binary release 2.10-0.8.2.1 . But we have been needing a encrypted communication between our publishers and subscribers. So we got 2.10-0.9.0.0. This works very well with no SSL enabled. But currently have issues with SSL enabled.

Re: Kafka SSL Configuration Problems

Please use advertised.listeners instead of advertised.host.name. See this comment: https://github.com/apache/kafka/pull/793#issuecomment-174287124 Ismael On Mon, Feb 1, 2016 at 4:44 PM, Nazario Parsacala wrote: > Hi, > > We were using kafka for a while now. We have been

Re: Kafka SSL Configuration Problems

Ok, This is getting interesting .. On the broker side, it is saying that it is registering 9092 as PLAINTEXT and 9093 as SSL [2016-02-01 13:26:33,796] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT),SSL ->

Re: Kafka SSL Configuration Problems

Hmm. So I removed port 9092 and just use port 9093. So no PLAINTEXT just SSL advertised.listeners=SSL://reactor.us.cixsoft.net:9093 Cleared Zookeeper and Kafka store and restart .. You see that it is registering 9093 onbly [2016-02-01 13:35:51,729] INFO Registered broker 0 at path

Fwd: Re: Kafka SSL Configuration Problems

Hello Nazario, Could you try it by creating a new topic? Thank you, Anirudh That works. At least it is saying that it is registering now with the SSL side. [2016-02-01 12:29:40,184] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT ->

Re: Kafka SSL Configuration Problems

No juice. /kafka-topics.sh --describe --topic anotherone --zookeeper localhost:2181 Topic:anotheronePartitionCount:4ReplicationFactor:1 Configs: Topic: anotherone Partition: 0Leader: 0 Replicas: 0 Isr: 0 Topic: anotherone Partition:

Re: Kafka SSL Configuration Problems

That works. At least it is saying that it is registering now with the SSL side. [2016-02-01 12:29:40,184] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT),SSL -> EndPoint(servername,9093,SSL) (kafka.utils.ZkUtils) Thank you.