SOLVED @Jose >If so, the review SSL conf related to that. It turned out that in the SSL configuration file, the `extendedKeyUsage` attribute was set to "serverAuth". So I extended it to "serverAuth, clientAuth" which solved the problem. At the moment it seems everything works as intended. The next step is to verify that every communication ( producers-Kafka cluster, Kafka inter-cluster, consumer-Kafka cluster, Zookeeper-Kafka, Zookeeper inter-cluster) is indeed secured.
Thank you for your help! On Mon, Oct 28, 2019 at 6:36 PM Jose Manuel Vega Monroy < jose.mon...@williamhill.com> wrote: > @Peter > > I have the feeling is related to client.auth required, in the end each > broker is a client for the rest in the cluster. > > Try with client.auth=none, and check if the connect. > > If so, the review SSL conf related to that. > > Cheers > > Get Outlook for Android <https://aka.ms/ghei36> > ------------------------------ > *From:* Péter Nagykátai <st4r.f1...@gmail.com> > *Sent:* Monday, October 28, 2019 2:47:38 PM > *To:* users@kafka.apache.org <users@kafka.apache.org> > *Subject:* Re: [EXTERNAL] SSL setup failing > > @Jose > >9092 is as well SSL protocol? > Yes, it is. As you see in the config snippet from my initial email. > > > Zookeeper is connecting over SSL? > Yes, at least as far as I can tell. It's set up there too but neither of > those are making verifying that easy... > > >So then I would review all certificates to check if valid. > I did that after your first response. > > >As well there is a Kafka broker property 'advertised.host.name' you could > set with same hostname in the certificate. > I added this property but didn't change anything, I get the exact same > error messages. > > Thanks > > On Mon, Oct 28, 2019 at 2:08 PM Jose Manuel Vega Monroy < > jose.mon...@williamhill.com> wrote: > > > @Peter > > > > 9092 is as well SSL protocol? Zookeeper is connecting over SSL? > > > > So then I would review all certificates to check if valid. > > > > As well there is a Kafka broker property 'advertised.host.name' you > could > > set with same hostname in the certificate. > > > > Thanks > > > > <http://www.williamhill.com/> > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.whenthefunstops.co.uk_&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=EpV0EqDiDgDPupfjZMqzuAv4qAvJraWxyVssyKeT39o&s=nltRImLIvceytcuboTTrAYca_JBmNpjbPGfTEcexHVw&e= > > > > Jose Manuel Vega Monroy > > Java Developer / Software Developer Engineer in Test > > Direct: +0035 0 2008038 (Ext. 8038) > > Email: jose.mon...@williamhill.com > > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA > > > > > > > > > > On 28/10/2019, 14:03, "Péter Nagykátai" <st4r.f1...@gmail.com> wrote: > > > > Sorry, if I was unclear before. I'm absolutely new to Kafka and how > it > > works. > > > > @Jose > > >That happening when clients trying to SSL connect? > > There are no clients at the moment just one Kafka broker which spews > > the > > errors in the server.log. To be specific, there is a ZooKeeper client > > which > > has no issues: > > >> INFO [ZooKeeperClient] Connected. > (kafka.zookeeper.ZooKeeperClient) > > > > @Manna > > >Are you talking about local network loopback? > > No, at least I don't think so. I'm simply trying to have the broker > in > > a > > stable running state but after it starts, it tries to connect to the > > listed > > internal broker, which is itself since at the moment there aren't > other > > brokers in the cluster. > > > > >Also, have you tried ssl debug using openssl? What did you observe? > > Per Jose's advice I checked the certificates I generated last week > and > > everything checked out on the 'rootCA' node. > > > > >How have you setup your signed certificates? > > I have a secured node where I generate certificates for every server > > in the > > cluster (with an intermediate CA). Here are the commands I used: > > > > `openssl genrsa -out kafka-1.key.pem 2048` > > `openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem > -new > > -sha256 -out kafka-1.csr.pem` > > `openssl ca -config openssl_intermediate.cnf -extensions server_cert > > -days > > 375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem` > > `openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem > > -out > > kafka-1.p12 -name kafka-1` > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype > PKCS12 > > -alias kafka-1 -destkeystore kafka-1.jks` > > > > Also, for the root+intermediate chain: > > `keytool -importcert -alias ca-root -keystore truststore.jks -file > > ca-chain.cert.pem` > > > > >Does your CN/SAN matches with your advertised.listeners setup? > > Yes. > > > > >Have you setup hostname verification correctly? > > My Kafka configuration file only have the settings I pasted before, > the > > rest aren't network specific. > > > > > > My (beginner) opinion is that Kafka tries to authenticate itself as a > > client and gets confused when getting 'server_hello' message. > > ("Unexpected > > handshake message: server_hello") > > > > Thanks! > > > > On Mon, Oct 28, 2019 at 12:25 PM M. Manna <manme...@gmail.com> > wrote: > > > > > Hi, > > > > > > not sure what it means "Tries to communicate with itself". Are you > > talking > > > about local network loopback? > > > > > > Also, have you tried ssl debug using openssl? What did you observe? > > > > > > The exception is handshake exception. This is quite common when > your > > cert > > > validation fails. How have you setup your signed certificates? Does > > your > > > CN/SAN matches with your advertised.listeners setup? Have you setup > > > hostname verification correctly? > > > > > > Thanks, > > > > > > On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai < > st4r.f1...@gmail.com> > > > wrote: > > > > > > > @Jose > > > > > > > > >It looks like communication problem between brokers. > > > > As I mentioned, "I can't get the first broker started". The > > message above > > > > is from when the broker tries to communicate with "itself": > > [Controller > > > > id=1001, targetBrokerId=1001]). > > > > > > > > Nevertheless, I went through the checklist and everything is in > > order. > > > For > > > > the first couple of tries, I got different SSL errors but I could > > work > > > > those out (that time I messed up the certificates), but now the > > problem > > > is: > > > > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected > > handshake > > > > **message: > > > > server_hello* > > > > > > > > Peter > > > > > > > > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy < > > > > jose.mon...@williamhill.com> wrote: > > > > > > > > > @Peter > > > > > > > > > > It looks like communication problem between brokers. But > ensure: > > > > > > > > > > 1) Crtificates are valid and properly signed by root CA or > > intermediate > > > > > one in the chain > > > > > 2) Clients and brokers having private key and certificate in > > their > > > > > keystore and properly configured to point to its path > > > > > 3) Clients and brokers having CA certificates in the truststore > > and > > > > > properly configured to point to its path > > > > > 4) Clients and brokersbroker having root CA certificate in > their > > > keystore > > > > > and properly configured to.point to its path > > > > > 5) Permissions are right ones fro trustore and keystore > > > > > > > > > > Thanks > > > > > > > > > > Get Outlook for Android < > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=LbcoJotfup6doDvIgqyHQISw77V3SVM4PTr_r6Tbn-Y&s=M2vFRWGBcJ0-qtIHW1BhAMr8yANCnvghejnBHBuTeJA&e= > > > > > > > > > > > > > ------------------------------ > > > > > *From:* Péter Nagykátai <st4r.f1...@gmail.com> > > > > > *Sent:* Monday, 28 October 2019, 00:13 > > > > > *To:* users@kafka.apache.org > > > > > *Subject:* [EXTERNAL] SSL setup failing > > > > > > > > > > Hi! > > > > > > > > > > I'm experimenting with setting up a log ingesting cluster and > > Kafka > > > would > > > > > be part of it. Unfortunately, I can't get the first broker > > started. I > > > > need > > > > > to secure the communication between a dozen nodes and Kaquiafka > > would > > > > only > > > > > be > > > > > one part of it. I have a secured node where I generate > > certificates for > > > > > every server in the cluster (with an intermediate CA). AFAIK, I > > need to > > > > use > > > > > '.jks' files for Kafka, so I've generated a '.p12' file from > the > > > openssl > > > > > certificate and key then used `keytool` to generate a keystore: > > > > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype > > PKCS12 > > > > > -alias kafka-1 -destkeystore kafka-1.jks` > > > > > I generated a truststore for the root and intermediate chain as > > well: > > > > > `keytool -importcert -alias ca-root -keystore truststore.jks > > -file > > > > > ca-chain.cert.pem > > > > > > > > > > Relevant part of the 'server.properties' configuration: > > > > > #### > > > > > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093 > > > > > > > advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093 > > > > > inter.broker.listener.name=INTERNAL > > > > > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL > > > > > security.protocol=SSL > > > > > ssl.client.auth=required > > > > > ssl.truststore.location=/*******/truststore.jks > > > > > ssl.truststore.password=************* > > > > > ssl.keystore.location=/*******/kafka-1.jks > > > > > ssl.keystore.password=************* > > > > > #### > > > > > > > > > > After starting Kafka (as a service) I get the the following in > > the > > > > > 'server.log': > > > > > >>... > > > > > >> INFO [KafkaServer id=1001] started > (kafka.server.KafkaServer) > > > > > >> INFO [SocketServer brokerId=1001] Failed authentication with > > > > > /XXX.XXX.XXX.XXX (SSL handshake failed) > > > > > (org.apache.kafka.common.network.Selector) > > > > > >> INFO [Controller id=1001, targetBrokerId=1001] Failed > > authentication > > > > > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed) > > > > > (org.apache.kafka.common.network.Selector) > > > > > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection > to > > node > > > > 1001 > > > > > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: > SSL > > > > handshake > > > > > failed (org.apache.kafka.clients.NetworkClient) > > > > > >>... > > > > > >> WARN SSL handshake failed (kafka.utils.CoreUtils$) > > > > > >> org.apache.kafka.common.errors.SslAuthenticationException: > SSL > > > > handshake > > > > > failed > > > > > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected > > handshake > > > > > message: server_hello > > > > > >>... > > > > > > > > > > I couldn't find any lead with that error message and got stuck. > > Any > > > ideas > > > > > what that error message means and how to solve it? > > > > > > > > > > Specs: > > > > > - Ubuntu 18.04.3 LTS > > > > > - OpenJDK Runtime Environment (build > > > > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3) > > > > > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz) > > > > > - OpenSSL 1.1.1 > > > > > > > > > > Thank you! > > > > > Peter > > > > > > > > > > Confidentiality: The contents of this e-mail and any > attachments > > > > > transmitted with it are intended to be confidential to the > > intended > > > > > recipient; and may be privileged or otherwise protected from > > > disclosure. > > > > If > > > > > you are not an intended recipient of this e-mail, do not > > duplicate or > > > > > redistribute it by any means. Please delete it and any > > attachments and > > > > > notify the sender that you have received it in error. This > > e-mail is > > > sent > > > > > by a William Hill PLC group company. The William Hill group > > companies > > > > > include, among others, William Hill PLC (registered number > > 4212563), > > > > > William Hill Organization Limited (registered number 278208), > > William > > > > Hill > > > > > US HoldCo Inc, WHG (International) Limited (registered number > > 99191) > > > and > > > > Mr > > > > > Green Limited (registered number C43260). Each of William Hill > > PLC and > > > > > William Hill Organization Limited is registered in England and > > Wales > > > and > > > > > has its registered office at 1 Bedford Avenue, London, WC1B > 3AU, > > UK. > > > > > William Hill U.S. HoldCo, Inc. is registered in Delaware and > has > > its > > > > > registered office at 1007 N. Orange Street, 9 Floor, > Wilmington, > > New > > > > Castle > > > > > County DE 19801 Delaware, United States of America. WHG > > (International) > > > > > Limited is registered in Gibraltar and has its registered > office > > at 6/1 > > > > > Waterport Place, Gibraltar. Mr Green Limited is registered in > > Malta and > > > > has > > > > > its registered office at Tagliaferro Business Centre, Level 7, > > 14 High > > > > > Street, Sliema SLM 1549, Malta. Unless specifically indicated > > > otherwise, > > > > > the contents of this e-mail are subject to contract; and are > not > > an > > > > > official statement, and do not necessarily represent the views, > > of > > > > William > > > > > Hill PLC, its subsidiaries or affiliated companies. Please note > > that > > > > > neither William Hill PLC, nor its subsidiaries and affiliated > > companies > > > > can > > > > > accept any responsibility for any viruses contained within this > > e-mail > > > > and > > > > > it is your responsibility to scan any emails and their > > attachments. > > > > William > > > > > Hill PLC, its subsidiaries and affiliated companies may monitor > > e-mail > > > > > traffic data and also the content of e-mails for effective > > operation of > > > > the > > > > > e-mail system, or for security, purposes. > > > > > > > > > > > > > > > > > > Confidentiality: The contents of this e-mail and any attachments > > transmitted with it are intended to be confidential to the intended > > recipient; and may be privileged or otherwise protected from disclosure. > If > > you are not an intended recipient of this e-mail, do not duplicate or > > redistribute it by any means. Please delete it and any attachments and > > notify the sender that you have received it in error. This e-mail is sent > > by a William Hill PLC group company. The William Hill group companies > > include, among others, William Hill PLC (registered number 4212563), > > William Hill Organization Limited (registered number 278208), William > Hill > > US HoldCo Inc, WHG (International) Limited (registered number 99191) and > Mr > > Green Limited (registered number C43260). Each of William Hill PLC and > > William Hill Organization Limited is registered in England and Wales and > > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. > > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its > > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New > Castle > > County DE 19801 Delaware, United States of America. WHG (International) > > Limited is registered in Gibraltar and has its registered office at 6/1 > > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and > has > > its registered office at Tagliaferro Business Centre, Level 7, 14 High > > Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, > > the contents of this e-mail are subject to contract; and are not an > > official statement, and do not necessarily represent the views, of > William > > Hill PLC, its subsidiaries or affiliated companies. Please note that > > neither William Hill PLC, nor its subsidiaries and affiliated companies > can > > accept any responsibility for any viruses contained within this e-mail > and > > it is your responsibility to scan any emails and their attachments. > William > > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail > > traffic data and also the content of e-mails for effective operation of > the > > e-mail system, or for security, purposes. > > > Confidentiality: The contents of this e-mail and any attachments > transmitted with it are intended to be confidential to the intended > recipient; and may be privileged or otherwise protected from disclosure. If > you are not an intended recipient of this e-mail, do not duplicate or > redistribute it by any means. Please delete it and any attachments and > notify the sender that you have received it in error. This e-mail is sent > by a William Hill PLC group company. The William Hill group companies > include, among others, William Hill PLC (registered number 4212563), > William Hill Organization Limited (registered number 278208), William Hill > US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr > Green Limited (registered number C43260). Each of William Hill PLC and > William Hill Organization Limited is registered in England and Wales and > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle > County DE 19801 Delaware, United States of America. WHG (International) > Limited is registered in Gibraltar and has its registered office at 6/1 > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has > its registered office at Tagliaferro Business Centre, Level 7, 14 High > Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, > the contents of this e-mail are subject to contract; and are not an > official statement, and do not necessarily represent the views, of William > Hill PLC, its subsidiaries or affiliated companies. Please note that > neither William Hill PLC, nor its subsidiaries and affiliated companies can > accept any responsibility for any viruses contained within this e-mail and > it is your responsibility to scan any emails and their attachments. William > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail > traffic data and also the content of e-mails for effective operation of the > e-mail system, or for security, purposes. >
