Hi Jens,
Thanks for your quick reply. After double checking certificate extended key
usage, it looks they are configured as required:
openssl x509 –in full-nifi-node1.pem –noout –text
…
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
…
Emmanuel
Emmanuel Quevillon
DevOps | Safran Analytics
• M +33 (0)6 03 44 32 86
SAFRAN Paris-Saclay
Rue des Jeunes Bois
Châteaufort
CS 80112
78772 Magny-Les-Hameaux
www.safran-group.com<http://www.safran-group.com/fr/>
[SAFRAN]
C2 - Restricted
De : Jens M. Kofoed <jmkofoed....@gmail.com>
Envoyé : mercredi 27 octobre 2021 15:40
À : users@nifi.apache.org
Objet : Re: Nifi secured cluster can't send heartbeat between nodes
Hi
I don’t know if this will help. But the certificate used by nifi needs be both
a server auth and client auth. Normally certificates are only one of them. The
nifi certificate use the server auth for the web ui and when other servers
connect to it. It is using the client auth when nifi talks to other nifi
servers.
Regards
Jens
Den 27. okt. 2021 kl. 15.27 skrev QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING
SERVICES (SAFRAN)
<emmanuel.quevillon.e...@safrangroup.com<mailto:emmanuel.quevillon.e...@safrangroup.com>>:
Hi list,
I’m facing a weird problem I can’t resolve or even understand with my secured
nifi cluster. Below is the situation.
We have setup a secured nifi cluster with 3 nodes, say node1,node2 and node3.
For each of theses nodes, we’ve manually created a SSL certificate signing
request (CSR) (using a password protected private key) to be signed by our
internal CA.
Once we’ve get the certificates signed, I’ve installed each node certificates
following this procedure:
1) Add the full certificate chain (root + intermediate certificates) into
the signed certificate.
cat nifi-nodeX.pem cert_chain.pem > full-nifi-nodeX.pem
2) Create a PKCS12 certificate using private key (.key) and full signed
certificate (.pem)
openssl pkcs12 -export -in full-nifi-nodeX.pem -inkey nifi-nodeX.key -out
nifi-nodeX.p12 \
-name nifi-nodeX -passin pass:"XXXXXX" -passout pass:YYYYY;
3) Import nifi-nodeX.p12 into the nifi-nodeX keystore
keytool –omportkeystore –deststorepass xxxxxx –destkeystore keystore.jks
–srckeystore nifi-nodeX.p12 –srcstoretype PKCS12
4) Then added each other nifi-node certificates (.pem) into nifi-truststore
node1: add full-nifi-node2 + full-nifi-node3 into truststore
node2: add full-nifi-node1 + full-nifi-node3 into truststore
node3: add full-nifi-node2 + full-nifi-node1 into truststore
5) Restarted each node
Once each node are restarted, I can connect to the web UI, but I’ve got an
error message saying:
For info, web UI is reachable on port 8443
Invalid State:
The Flow Controller is initializing the Data Flow.
Looking at node logs (nifi-app.log) I can see that each node cannot talk to
each other and to the Coordinator to send heartbeat messages:
Nifi-node1:
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that
Cluster Coordinator is located at nifi-node2:11443; will use this address for
sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located
at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to
cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed
unmarshalling 'CONNECTION_RESPONSE' protocol message from
nifi-node2.rd1.rf1/10.108.70.39:11443 due to:
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
…
Nifi-node2:
WARN [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener
Failed processing protocol message from nifi-node3 due to Empty client
certificate chain
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that
Cluster Coordinator is located at nifi-node2:11443; will use this address for
sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located
at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to
cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed
marshalling 'CONNECTION_REQUEST' protocol message due to:
java.net.SocketException: Connection reset by peer (Write failed)
…
Nifi-node3:
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that
Cluster Coordinator is located at nifi-node2:11443; will use this address for
sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located
at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to
cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed
unmarshalling 'CONNECTION_RESPONSE' protocol message from
nifi-node2/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException:
Received fatal alert: bad_certificate
It looks like the signed certificates are not ok regarding the logs errors.
However, trying these certificates using openssl s_client command works as
expected:
openssl s_client -connect nifi-node3:11443 -cert full-nifi-node3.pem -key
nifi-node3.key -pass pass:'XXXXXXX’
CONNECTED(00000003)
depth=3 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Root CA 1
verify return:1
depth=2 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate CA 1
verify return:1
depth=1 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate Service
CA 2
verify return:1
depth=0 C = FR, O = SAFRAN, OU = SAFRAN SA, OU = 0002 562082909, CN = nifi-node3
verify return:1
---
Certificate chain
0 s:/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
1 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1
2 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
3 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
---
Server certificate
-----BEGIN CERTIFICATE-----
….
-----END CERTIFICATE-----
subject=/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
issuer=/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
---
Acceptable client certificate CA names
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=niif-node2
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node1
/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=Safran Nifi Admin
/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=localhost
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms:
0x07+0x08:0x08+0x08:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x04+0x08:0x05+0x08:0x06+0x08:0x09+0x08:0x0A+0x08:0x0B+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms:
ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 8911 bytes and written 8534 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 68686A816F510BED151FEBB80604862B799CD0D5DFCEA9602A9E204E9EC5741E
Session-ID-ctx:
Master-Key:
CB4E24EDCAA3518494C04762965452CDC9CE993FCCAF3DBCCF76755376B808667342AF327DE5B8DE6B3B981F55B3CB90
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1635340626
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
closed
There is something I don’t get!
I’ve tried all the above procedure without adding the full cert chain, same
errors.
I’ve tried with autogenerated self-signed certificates using nifi-toolkit, and
it works as expected, so I think there is definitely something wrong with the
signed certificates but I’ve no clue at all what it could be.
Please could someone light my lantern, I’ve no more idea or way to explore.
Regards
Emmanuel
C2 - Restricted
#
" Ce courriel et les documents qui lui sont joints peuvent contenir des
informations confidentielles, être soumis aux règlementations relatives au
contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas
destinés, nous vous signalons qu'il est strictement interdit de les divulguer,
de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu.
Toute exportation ou réexportation non autorisée est interdite Si ce message
vous a été transmis par erreur, merci d'en informer l'expéditeur et de
supprimer immédiatement de votre système informatique ce courriel ainsi que
tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or
proprietary information and may be subject to export control laws and
regulations. If you are not the intended recipient, you are notified that any
dissemination, copying of this e-mail and any attachments thereto or use of
their contents by any means whatsoever is strictly prohibited. Unauthorized
export or re-export is prohibited. If you have received this e-mail in error,
please advise the sender immediately and delete this e-mail and all attached
documents from your computer system."
#
#
" Ce courriel et les documents qui lui sont joints peuvent contenir des
informations confidentielles, être soumis aux règlementations relatives au
contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas
destinés, nous vous signalons qu'il est strictement interdit de les divulguer,
de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu.
Toute exportation ou réexportation non autorisée est interdite Si ce message
vous a été transmis par erreur, merci d'en informer l'expéditeur et de
supprimer immédiatement de votre système informatique ce courriel ainsi que
tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or
proprietary information and may be subject to export control laws and
regulations. If you are not the intended recipient, you are notified that any
dissemination, copying of this e-mail and any attachments thereto or use of
their contents by any means whatsoever is strictly prohibited. Unauthorized
export or re-export is prohibited. If you have received this e-mail in error,
please advise the sender immediately and delete this e-mail and all attached
documents from your computer system."
#
