Hi Ryan, Sorry you are encountering problems. You’ve clearly done research on the issue and we always want to make this process easier and more straightforward.
The user log is showing you can successfully authenticate with the client
certificate. I have a couple questions:
* Are you trying to run multiple nodes on the same IP address?
* What is the output of examining the keytool command against the
conf/keystore.jks?
* Does the certificate used for NiFi contain the exact hostname as the
CN? An IP address? Any SAN entries?
* Are you running NiFi as root? Starting a process to listen on 443 (or any
port below 1024) requires root.
* Is the cluster port unique on each node?
* Did you run the TLS toolkit once to generate 3 unique certificates and
configurations, 3 separate times, or once with one certificate generated which
you shared across multiple nodes?
* Are the ports available in whatever AWS firewall you’re using?
From the stacktrace, it appears the issue is during the request replication
from the node whose UI you are hitting to the other nodes in the cluster. The
“HTTPS hostname wrong: should be” message is almost always related to the
certificate hostname not matching the host/IP that the service is running.
Can you try this command from the terminal on a node that works (“node 1”) to
another node (“node 2”)?
* host - the hostname/IP of “node 2”
* port - the port of “node 2”
* path_to_your_cert - the certificate containing the public key of the client
key pair in PEM encoding (see command below)
* path_to_your_key - the client private key in PEM encoding
* path_to_your_CA_cert - the certificate containing the public key of the CA
(for server certificate path verification) in PEM encoding
$ openssl s_client -connect <host:port> -debug -state -cert
<path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
<path_to_your_CA_cert.pem>
The response from this should be extensive, but the interesting portion is the
end — you should see the actual session keys that are negotiated during the TLS
handshake and a status code:
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: C0DAFA...162F56
Session-ID-ctx:
Master-Key: 1ED6A4...3358AC
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 100800 (seconds)
TLS session ticket:
0000 - 61 b0 a6 81 81 61 19 77-c2 d5 d9 43 99 07 94 2f a....a.w...C.../
...
00a0 - 1e 3c b2 8d .<..
Start Time: 1488378535
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
You want a “0” result; other common response codes include 19 (self-signed
certificate in the chain) or 21 (unable to verify first certificate). If you
can provide that output, that may also help us diagnose this issue.
(To export the public and private keys from the PKCS12 client keystore):
hw12203:/Users/alopresto/Workspace/scratch/openiot (master) alopresto
🔓 17902s @ 22:05:34 $ def export_client
export_client ()
{
echo "Exporting client certificate and key from $1";
read -s -p "Keystore Password: " PASSWORD;
openssl pkcs12 -in "$1" -out client.der -nodes -password "pass:$PASSWORD";
openssl pkcs12 -in "$1" -nodes -nocerts -out client.key -password
"pass:$PASSWORD";
unset PASSWORD;
if [ "$(grep -e '-----BEGIN CERTIFICATE-----' client.der -F -c)" -gt 0 ];
then
perl -pe 'BEGIN{undef $/;} s|-----BEGIN PRIVATE KEY-----.*?-----END
PRIVATE KEY-----|Removed private key|gs' client.der > client.pem;
else
openssl x509 -inform der -in client.der -out client.pem;
fi;
ls -alGFh client.*
}
hw12203:/Users/alopresto/Workspace/scratch/openiot (master) alopresto
🔓 652581s @ 09:23:34 $ export_client client.p12
Side note: You will probably want to update the passwords for your keystore and
truststore after including them in the email below.
Andy LoPresto
[email protected]
[email protected]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Mar 1, 2017, at 8:43 AM, Ryan H <[email protected]> wrote:
>
> Dear NiFi Dev Team,
>
> I am currently in the process of setting up a secure NiFi Cluster and running
> into an issue that I cannot resolve (almost a couple weeks now)--any help
> will be greatly appreciated.
>
> As background, I have followed these articles (among others) to set up the
> secure cluster:
> https://community.hortonworks.com/articles/886/securing-nifi-step-by-step.html
>
> <https://community.hortonworks.com/articles/886/securing-nifi-step-by-step.html>
> http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>
> <http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy>
>
> A thread that I read thru and followed to try and correct the issue (which
> looks to be the same error I am getting, but does not resolve the problem):
> http://apache-nifi.1125220.n5.nabble.com/Clustering-configuration-error-HTTPS-hostname-wrong-td13680.html
>
> <http://apache-nifi.1125220.n5.nabble.com/Clustering-configuration-error-HTTPS-hostname-wrong-td13680.html>
>
> What I have been able to do:
> I have been able to set up a secure cluster of 3 nodes on my local machine
> (using the NiFI Toolkit for cert/truststore/keystore generation).
> I have been able to set up a standalone (not clustered) secure NiFi instance
> in my target environment (AWS farm).
> I have been able to set up an unsecured cluster in my target environment (the
> same AWS farm).
>
> What I have not been able to do (which is what I am trying to accomplish):
> I have not been able to setup a secure cluster in my target environment. I
> have started with just a single machine in my AWS farm with a single node
> (same machine I was able to set up a secure standalone instance of NiFI). I
> have been trying to get just a single node as a secure cluster which I would
> then replicate across the other machines in the farm once successful.
>
> I have tried various methods: NiFi Toolkit, TinyCert, and self signed
> certificates which I have generated by hand via openssl.
>
> The current certs/truststore/keystore have all been generated using the NiFi
> Toolkit version 1.1.1 and the version of NiFi I am running is 1.1.1
>
> Below are my relevant config settings and the resultant log output which is
> the point at which I am stuck. Any help is very much appreciated. As an FYI,
> everywhere below that says "my.ip.address" is the actual IP address of my AWS
> server.
>
> Thanks in Advance,
>
> Ryan H.
>
>
>
>
> "nifi.properties"
>
> # Site to Site properties
> nifi.remote.input.host=my.ip.address
> nifi.remote.input.secure=true
> nifi.remote.input.socket.port=10443
> nifi.remote.input.http.enabled=true
> nifi.remote.input.http.transaction.ttl=30 sec
>
> # web properties #
> nifi.web.war.directory=./lib
> nifi.web.http.host=
> nifi.web.http.port=
> nifi.web.https.host=my.ip.address
> nifi.web.https.port=443
> nifi.web.jetty.working.directory=./work/jetty
> nifi.web.jetty.threads=200
>
> # security properties #
> nifi.sensitive.props.key=
> nifi.sensitive.props.key.protected=
> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
> nifi.sensitive.props.provider=BC
> nifi.sensitive.props.additional.keys=
>
> nifi.security.keystore=./conf/keystore.jks
> nifi.security.keystoreType=jks
> nifi.security.keystorePasswd=SiPE4sd4he5pI0kvmifB5DNnGTp/nnfGFVLO/TsVsIs
> nifi.security.keyPasswd=SiPE4sd4he5pI0kvmifB5DNnGTp/nnfGFVLO/TsVsIs
> nifi.security.truststore=./conf/truststore.jks
> nifi.security.truststoreType=jks
> nifi.security.truststorePasswd=Hs4tnGAAnywGF2Xo2+EKaJGlALMZcpJkzbZ/gdYcB2s
> nifi.security.needClientAuth=
> nifi.security.user.authorizer=file-provider
> nifi.security.user.login.identity.provider=
> nifi.security.ocsp.responder.url=
> nifi.security.ocsp.responder.certificate=
>
> # cluster common properties (all nodes must have same values) #
> nifi.cluster.protocol.heartbeat.interval=5 sec
> nifi.cluster.protocol.is.secure=true
>
> # cluster node properties (only configure for cluster nodes) #
> nifi.cluster.is.node=true
> nifi.cluster.node.address=my.ip.address
> nifi.cluster.node.protocol.port=11443
> nifi.cluster.node.protocol.threads=10
> nifi.cluster.node.event.history.size=25
> nifi.cluster.node.connection.timeout=5 sec
> nifi.cluster.node.read.timeout=5 sec
> nifi.cluster.firewall.file=
> nifi.cluster.flow.election.max.wait.time=2 mins
> nifi.cluster.flow.election.max.candidates=
>
> # zookeeper properties, used for cluster management #
> nifi.zookeeper.connect.string=my.ip.address:2181
> nifi.zookeeper.connect.timeout=3 secs
> nifi.zookeeper.session.timeout=3 secs
> nifi.zookeeper.root.node=/nifi
>
> -------------------------------------------------
>
> "authorizers.xml"
>
> <authorizer>
> <identifier>file-provider</identifier>
> <class>org.apache.nifi.authorization.FileAuthorizer</class>
> <property name="Authorizations
> File">./conf/authorizations.xml</property>
> <property name="Users File">./conf/users.xml</property>
> <property name="Initial Admin Identity">CN=admin, OU=NIFI</property>
> <property name="Legacy Authorized Users File"></property>
>
> <property name="Node Identity 1">CN=my.ip.address, OU=NIFI</property>
> <property name="Node Identity 2">CN=my.ip.address, OU=NIFI</property>
> <property name="Node Identity 3">CN=my.ip.address, OU=NIFI</property>
> </authorizer>
>
> -------------------------------------------------
>
> Output of ‘keytool -list -keystore
> nifi-toolkit-1.1.1/target/CN\=admin_OU\=NIFI.p12 -storepass <password>
> -storetype PKCS12 -v’
>
> Keystore type: PKCS12
> Keystore provider: SunJSSE
>
> Your keystore contains 1 entry
>
> Alias name: nifi-key
> Creation date: Feb 28, 2017
> Entry type: PrivateKeyEntry
> Certificate chain length: 2
> Certificate[1]:
> Owner: CN=admin, OU=NIFI
> Issuer: CN=localhost, OU=NIFI
> Serial number: 15a8674765e00000000
> Valid from: Tue Feb 28 20:41:03 UTC 2017 until: Fri Feb 28 20:41:03 UTC 2020
> Certificate fingerprints:
> MD5: ED:2A:AD:BD:88:C4:B8:49:80:EA:58:A2:A4:83:FD:67
> SHA1: 19:2F:3C:41:43:B0:A7:3C:C8:4D:1F:A6:11:DC:FA:EC:61:C0:AC:6B
> SHA256:
> 85:9F:E3:E6:2B:94:60:48:55:0B:5F:BF:55:DF:FC:C8:E8:30:CD:85:3B:2B:F6:9A:B9:56:5C:EC:2F:43:C8:42
> Signature algorithm name: SHA256withRSA
> Version: 3
>
> Extensions:
>
> #1: ObjectId: 2.5.29.35 Criticality=false
> AuthorityKeyIdentifier [
> KeyIdentifier [
> 0000: E1 FC AB 2F 69 4D 6A FE A5 E8 2F B6 43 13 79 1C .../iMj.../.C.y.
> 0010: 46 F5 23 91 F.#.
> ]
> ]
>
> #2: ObjectId: 2.5.29.19 Criticality=false
> BasicConstraints:[
> CA:false
> PathLen: undefined
> ]
>
> #3: ObjectId: 2.5.29.37 Criticality=false
> ExtendedKeyUsages [
> clientAuth
> serverAuth
> ]
>
> #4: ObjectId: 2.5.29.15 Criticality=true
> KeyUsage [
> DigitalSignature
> Non_repudiation
> Key_Encipherment
> Data_Encipherment
> Key_Agreement
> ]
>
> #5: ObjectId: 2.5.29.14 Criticality=false
> SubjectKeyIdentifier [
> KeyIdentifier [
> 0000: 69 E2 30 6B 51 4D 8F 5F 9D 96 B1 A2 17 64 AC 72 i.0kQM._.....d.r
> 0010: A6 37 0E 9F .7..
> ]
> ]
>
> Certificate[2]:
> Owner: CN=localhost, OU=NIFI
> Issuer: CN=localhost, OU=NIFI
> Serial number: 15a8674717300000000
> Valid from: Tue Feb 28 20:41:02 UTC 2017 until: Fri Feb 28 20:41:02 UTC 2020
> Certificate fingerprints:
> MD5: 83:47:1D:17:85:3E:EC:5D:7B:B8:75:24:C3:F8:86:75
> SHA1: 2C:F1:71:7D:2F:67:B7:C6:C5:28:11:EC:A3:F2:51:D7:B3:07:F9:92
> SHA256:
> 7C:63:31:EB:4A:E7:B1:26:C1:F5:67:89:46:90:EE:9C:5E:DD:B5:47:45:4A:FB:F9:06:0F:81:9B:3B:CB:F4:E7
> Signature algorithm name: SHA256withRSA
> Version: 3
>
> Extensions:
>
> #1: ObjectId: 2.5.29.35 Criticality=false
> AuthorityKeyIdentifier [
> KeyIdentifier [
> 0000: E1 FC AB 2F 69 4D 6A FE A5 E8 2F B6 43 13 79 1C .../iMj.../.C.y.
> 0010: 46 F5 23 91 F.#.
> ]
> ]
>
> #2: ObjectId: 2.5.29.19 Criticality=false
> BasicConstraints:[
> CA:true
> PathLen:2147483647
> ]
>
> #3: ObjectId: 2.5.29.37 Criticality=false
> ExtendedKeyUsages [
> clientAuth
> serverAuth
> ]
>
> #4: ObjectId: 2.5.29.15 Criticality=true
> KeyUsage [
> DigitalSignature
> Non_repudiation
> Key_Encipherment
> Data_Encipherment
> Key_Agreement
> Key_CertSign
> Crl_Sign
> ]
>
> #5: ObjectId: 2.5.29.14 Criticality=false
> SubjectKeyIdentifier [
> KeyIdentifier [
> 0000: E1 FC AB 2F 69 4D 6A FE A5 E8 2F B6 43 13 79 1C .../iMj.../.C.y.
> 0010: 46 F5 23 91 F.#.
> ]
> ]
>
> *******************************************
> *******************************************
>
> Output of "nifi-app.log"
>
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to my.ip.address:443 due to {}
> com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: HTTPS
> hostname wrong: should be <my.ip.address>
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
> ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.Client.handle(Client.java:652)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
> ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
> ~[jersey-client-1.19.jar:1.19]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.io.IOException: HTTPS hostname wrong: should be
> <my.ip.address>
> at
> sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649)
> ~[na:1.8.0_121]
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:573)
> ~[na:1.8.0_121]
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> ~[na:1.8.0_121]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
> ~[na:1.8.0_121]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> ~[na:1.8.0_121]
> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> ~[na:1.8.0_121]
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
> ~[na:1.8.0_121]
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
> ~[jersey-client-1.19.jar:1.19]
> ... 12 common frames omitted
>
> -------------------------------------------------
>
> Output of "nifi-user.log"
>
> 2017-02-28 21:03:47,763 INFO [NiFi Web Server-19]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Kerberos ticket login not supported by this NiFi.. Returning Conflict
> response.
> 2017-02-28 21:03:47,917 INFO [NiFi Web Server-16]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=admin, OU=NIFI)
> GET https://1my.ip.address/nifi-api/flow/current-user
> <https://1my.ip.address/nifi-api/flow/current-user> (source ip: my.machine.ip)
> 2017-02-28 21:03:47,919 INFO [NiFi Web Server-16]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=admin,
> OU=NIFI
> 2017-02-28 21:05:44,793 INFO [NiFi Web Server-22]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Kerberos ticket login not supported by this NiFi.. Returning Conflict
> response.
> 2017-02-28 21:05:44,892 INFO [NiFi Web Server-20]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=admin, OU=NIFI)
> GET https://my.ip.address/nifi-api/flow/current-user
> <https://my.ip.address/nifi-api/flow/current-user> (source ip: my.machine.ip)
> 2017-02-28 21:05:44,893 INFO [NiFi Web Server-20]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=admin,
> OU=NIFI
>
>
signature.asc
Description: Message signed with OpenPGP using GPGMail
