Hi Andy, Thanks you as well for a quick response.
Your Questions: * Are you trying to run multiple nodes on the same IP address? -Yes, for the purpose of the initial set up and "proof of concept" I am running more than one NiFi instance on the same machine (and same IP address). Once that was working for me, I would move to multiple machines (ideally with a single node on each machine which I believe is what is recommended) * What is the output of examining the keytool command against the conf/keystore.jks? * Does the certificate used for NiFi contain the exact hostname as the CN? An IP address? Any SAN entries? -It did have the same CN, but it was an IP address and not the hostname which looked to be the root of the problem. -No SAN entries, but the ability to add them with the NiFi-Toolkit will surely be beneficial in the next stages of my development. * Are you running NiFi as root? Starting a process to listen on 443 (or any port below 1024) requires root. -Yes * Is the cluster port unique on each node? -Yes, each node is configured with different ports. (80, 81, 82, etc) * Did you run the TLS toolkit once to generate 3 unique certificates and configurations, 3 separate times, or once with one certificate generated which you shared across multiple nodes? -I've tried pretty much each permutation of this. The most recent was running the following commands: When trying to cluster 3 nodes on same machine: ./bin/tls-toolkit.sh standalone -n 'my.ip.address(3)' -C 'CN=admin, OU=NIFI' -o './mydir' When trying for just a single node as a cluster: ./bin/tls-toolkit.sh standalone -n 'my.ip.address' -C 'CN=admin, OU=NIFI' -o './mydir' * Are the ports available in whatever AWS firewall you’re using? -Yes, this is set up correctly as far as I can tell. Bryan B responded just a few minutes before your response and he suggested that using an IP address may not work; which looked to be the root of the problem. I instead used the servers hostname instead and I got past the problem. I said that I would follow up with my full solution once everything seems to be stable so any other members that may run in to the issue have another documented solution. Thanks for the help and direction! I will follow up in the next day with my solution. Cheers, Ryan H. On Wed, Mar 1, 2017 at 9:36 AM, Andy LoPresto <[email protected]> wrote: > Hi Ryan, > > Sorry you are encountering problems. You’ve clearly done research on the > issue and we always want to make this process easier and more > straightforward. > > The user log is showing you can successfully authenticate with the client > certificate. I have a couple questions: > > * Are you trying to run multiple nodes on the same IP address? > * What is the output of examining the keytool command against the > conf/keystore.jks? > * Does the certificate used for NiFi contain the exact hostname as the CN? > An IP address? Any SAN entries? > * Are you running NiFi as root? Starting a process to listen on 443 (or > any port below 1024) requires root. > * Is the cluster port unique on each node? > * Did you run the TLS toolkit once to generate 3 unique certificates and > configurations, 3 separate times, or once with one certificate generated > which you shared across multiple nodes? > * Are the ports available in whatever AWS firewall you’re using? > > From the stacktrace, it appears the issue is during the request > replication from the node whose UI you are hitting to the other nodes in > the cluster. The “HTTPS hostname wrong: should be” message is almost always > related to the certificate hostname not matching the host/IP that the > service is running. > > Can you try this command from the terminal on a node that works (“node 1”) > to another node (“node 2”)? > > * host - the hostname/IP of “node 2” > * port - the port of “node 2” > * path_to_your_cert - the certificate containing the public key of the > client key pair in PEM encoding (see command below) > * path_to_your_key - the client private key in PEM encoding > * path_to_your_CA_cert - the certificate containing the public key of the > CA (for server certificate path verification) in PEM encoding > > $ openssl s_client -connect <host:port> -debug -state -cert > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile > <path_to_your_CA_cert.pem> > > The response from this should be extensive, but the interesting portion is > the end — you should see the actual session keys that are negotiated during > the TLS handshake and a status code: > > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256 > Server public key is 2048 bit > Secure Renegotiation IS supported > Compression: NONE > Expansion: NONE > No ALPN negotiated > SSL-Session: > Protocol : TLSv1.2 > Cipher : ECDHE-RSA-AES128-GCM-SHA256 > Session-ID: C0DAFA...162F56 > Session-ID-ctx: > Master-Key: 1ED6A4...3358AC > Key-Arg : None > PSK identity: None > PSK identity hint: None > SRP username: None > TLS session ticket lifetime hint: 100800 (seconds) > TLS session ticket: > 0000 - 61 b0 a6 81 81 61 19 77-c2 d5 d9 43 99 07 94 2f > a....a.w...C.../ > ... > 00a0 - 1e 3c b2 8d .<.. > > Start Time: 1488378535 > Timeout : 300 (sec) > Verify return code: 0 (ok) > --- > > You want a “0” result; other common response codes include 19 (self-signed > certificate in the chain) or 21 (unable to verify first certificate). If > you can provide that output, that may also help us diagnose this issue. > > (To export the public and private keys from the PKCS12 client keystore): > > hw12203:/Users/alopresto/Workspace/scratch/openiot (master) alopresto > 🔓 17902s @ 22:05:34 $ def export_client > export_client () > { > echo "Exporting client certificate and key from $1"; > read -s -p "Keystore Password: " PASSWORD; > openssl pkcs12 -in "$1" -out client.der -nodes -password > "pass:$PASSWORD"; > openssl pkcs12 -in "$1" -nodes -nocerts -out client.key -password > "pass:$PASSWORD"; > unset PASSWORD; > if [ "$(grep -e '-----BEGIN CERTIFICATE-----' client.der -F -c)" -gt 0 > ]; then > perl -pe 'BEGIN{undef $/;} s|-----BEGIN PRIVATE > KEY-----.*?-----END PRIVATE KEY-----|Removed private key|gs' client.der > > client.pem; > else > openssl x509 -inform der -in client.der -out client.pem; > fi; > ls -alGFh client.* > } > hw12203:/Users/alopresto/Workspace/scratch/openiot (master) alopresto > 🔓 652581s @ 09:23:34 $ export_client client.p12 > > *Side note: *You will probably want to update the passwords for your > keystore and truststore after including them in the email below. > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Mar 1, 2017, at 8:43 AM, Ryan H <[email protected]> > wrote: > > Dear NiFi Dev Team, > > I am currently in the process of setting up a secure NiFi Cluster and > running into an issue that I cannot resolve (almost a couple weeks > now)--any help will be greatly appreciated. > > As background, I have followed these articles (among others) to set up the > secure cluster: > https://community.hortonworks.com/articles/886/securing- > nifi-step-by-step.html > http://bryanbende.com/development/2016/08/17/apache- > nifi-1-0-0-authorization-and-multi-tenancy > > A thread that I read thru and followed to try and correct the issue (which > looks to be the same error I am getting, but does not resolve the problem): > http://apache-nifi.1125220.n5.nabble.com/Clustering- > configuration-error-HTTPS-hostname-wrong-td13680.html > > What I have been able to do: > I have been able to set up a secure cluster of 3 nodes on my local machine > (using the NiFI Toolkit for cert/truststore/keystore generation). > I have been able to set up a standalone (not clustered) secure NiFi > instance in my target environment (AWS farm). > I have been able to set up an unsecured cluster in my target environment > (the same AWS farm). > > What I have not been able to do (which is what I am trying to accomplish): > I have not been able to setup a secure cluster in my target environment. I > have started with just a single machine in my AWS farm with a single node > (same machine I was able to set up a secure standalone instance of NiFI). I > have been trying to get just a single node as a secure cluster which I > would then replicate across the other machines in the farm once successful. > > I have tried various methods: NiFi Toolkit, TinyCert, and self signed > certificates which I have generated by hand via openssl. > > The current certs/truststore/keystore have all been generated using the > NiFi Toolkit version 1.1.1 and the version of NiFi I am running is 1.1.1 > > Below are my relevant config settings and the resultant log output which > is the point at which I am stuck. Any help is very much appreciated. As an > FYI, everywhere below that says "my.ip.address" is the actual IP address of > my AWS server. > > Thanks in Advance, > > Ryan H. > > > > > "nifi.properties" > > # Site to Site properties > > > > nifi.remote.input.host=my.ip.address > nifi.remote.input.secure=true > nifi.remote.input.socket.port=10443 > nifi.remote.input.http.enabled=true > nifi.remote.input.http.transaction.ttl=30 sec > > # web properties # > > > > nifi.web.war.directory=./lib > nifi.web.http.host= > nifi.web.http.port= > nifi.web.https.host=my.ip.address > nifi.web.https.port=443 > nifi.web.jetty.working.directory=./work/jetty > nifi.web.jetty.threads=200 > > # security properties # > > > > nifi.sensitive.props.key= > nifi.sensitive.props.key.protected= > nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL > nifi.sensitive.props.provider=BC > nifi.sensitive.props.additional.keys= > > nifi.security.keystore=./conf/keystore.jks > nifi.security.keystoreType=jks > nifi.security.keystorePasswd=SiPE4sd4he5pI0kvmifB5DNnGTp/nnfGFVLO/TsVsIs > nifi.security.keyPasswd=SiPE4sd4he5pI0kvmifB5DNnGTp/nnfGFVLO/TsVsIs > nifi.security.truststore=./conf/truststore.jks > nifi.security.truststoreType=jks > nifi.security.truststorePasswd=Hs4tnGAAnywGF2Xo2+EKaJGlALMZcpJkzbZ/gdYcB2s > nifi.security.needClientAuth= > nifi.security.user.authorizer=file-provider > nifi.security.user.login.identity.provider= > nifi.security.ocsp.responder.url= > nifi.security.ocsp.responder.certificate= > > # cluster common properties (all nodes must have same values) > # > > > nifi.cluster.protocol.heartbeat.interval=5 sec > nifi.cluster.protocol.is.secure=true > > # cluster node properties (only configure for cluster nodes) > # > > > nifi.cluster.is.node=true > nifi.cluster.node.address=my.ip.address > nifi.cluster.node.protocol.port=11443 > nifi.cluster.node.protocol.threads=10 > nifi.cluster.node.event.history.size=25 > nifi.cluster.node.connection.timeout=5 sec > nifi.cluster.node.read.timeout=5 sec > nifi.cluster.firewall.file= > nifi.cluster.flow.election.max.wait.time=2 mins > nifi.cluster.flow.election.max.candidates= > > # zookeeper properties, used for cluster management > # > > > nifi.zookeeper.connect.string=my.ip.address:2181 > nifi.zookeeper.connect.timeout=3 secs > nifi.zookeeper.session.timeout=3 secs > nifi.zookeeper.root.node=/nifi > > ------------------------------------------------- > > "authorizers.xml" > > <authorizer> > <identifier>file-provider</identifier> > <class>org.apache.nifi.authorization.FileAuthorizer</class> > <property name="Authorizations File">./conf/authorizations. > xml</property> > <property name="Users File">./conf/users.xml</property> > <property name="Initial Admin Identity">CN=admin, > OU=NIFI</property> > <property name="Legacy Authorized Users File"></property> > > <property name="Node Identity 1">CN=my.ip.address, > OU=NIFI</property> > <property name="Node Identity 2">CN=my.ip.address, > OU=NIFI</property> > <property name="Node Identity 3">CN=my.ip.address, > OU=NIFI</property> > </authorizer> > > ------------------------------------------------- > > Output of ‘keytool -list -keystore > nifi-toolkit-1.1.1/target/CN\=admin_OU\=NIFI.p12 > -storepass <password> -storetype PKCS12 -v’ > > Keystore type: PKCS12 > Keystore provider: SunJSSE > > Your keystore contains 1 entry > > Alias name: nifi-key > Creation date: Feb 28, 2017 > Entry type: PrivateKeyEntry > Certificate chain length: 2 > Certificate[1]: > Owner: CN=admin, OU=NIFI > Issuer: CN=localhost, OU=NIFI > Serial number: 15a8674765e00000000 > Valid from: Tue Feb 28 20:41:03 UTC 2017 until: Fri Feb 28 20:41:03 UTC > 2020 > Certificate fingerprints: > MD5: ED:2A:AD:BD:88:C4:B8:49:80:EA:58:A2:A4:83:FD:67 > SHA1: 19:2F:3C:41:43:B0:A7:3C:C8:4D:1F:A6:11:DC:FA:EC:61:C0:AC:6B > SHA256: 85:9F:E3:E6:2B:94:60:48:55:0B:5F:BF:55:DF:FC:C8:E8:30:CD:85: > 3B:2B:F6:9A:B9:56:5C:EC:2F:43:C8:42 > Signature algorithm name: SHA256withRSA > Version: 3 > > Extensions: > > #1: ObjectId: 2.5.29.35 Criticality=false > AuthorityKeyIdentifier [ > KeyIdentifier [ > 0000: E1 FC AB 2F 69 4D 6A FE A5 E8 2F B6 43 13 79 1C .../iMj.../.C.y. > 0010: 46 F5 23 91 F.#. > ] > ] > > #2: ObjectId: 2.5.29.19 Criticality=false > BasicConstraints:[ > CA:false > PathLen: undefined > ] > > #3: ObjectId: 2.5.29.37 Criticality=false > ExtendedKeyUsages [ > clientAuth > serverAuth > ] > > #4: ObjectId: 2.5.29.15 Criticality=true > KeyUsage [ > DigitalSignature > Non_repudiation > Key_Encipherment > Data_Encipherment > Key_Agreement > ] > > #5: ObjectId: 2.5.29.14 Criticality=false > SubjectKeyIdentifier [ > KeyIdentifier [ > 0000: 69 E2 30 6B 51 4D 8F 5F 9D 96 B1 A2 17 64 AC 72 i.0kQM._.....d.r > 0010: A6 37 0E 9F .7.. > ] > ] > > Certificate[2]: > Owner: CN=localhost, OU=NIFI > Issuer: CN=localhost, OU=NIFI > Serial number: 15a8674717300000000 > Valid from: Tue Feb 28 20:41:02 UTC 2017 until: Fri Feb 28 20:41:02 UTC > 2020 > Certificate fingerprints: > MD5: 83:47:1D:17:85:3E:EC:5D:7B:B8:75:24:C3:F8:86:75 > SHA1: 2C:F1:71:7D:2F:67:B7:C6:C5:28:11:EC:A3:F2:51:D7:B3:07:F9:92 > SHA256: 7C:63:31:EB:4A:E7:B1:26:C1:F5:67:89:46:90:EE:9C:5E:DD:B5:47: > 45:4A:FB:F9:06:0F:81:9B:3B:CB:F4:E7 > Signature algorithm name: SHA256withRSA > Version: 3 > > Extensions: > > #1: ObjectId: 2.5.29.35 Criticality=false > AuthorityKeyIdentifier [ > KeyIdentifier [ > 0000: E1 FC AB 2F 69 4D 6A FE A5 E8 2F B6 43 13 79 1C .../iMj.../.C.y. > 0010: 46 F5 23 91 F.#. > ] > ] > > #2: ObjectId: 2.5.29.19 Criticality=false > BasicConstraints:[ > CA:true > PathLen:2147483647 <(214)%20748-3647> > ] > > #3: ObjectId: 2.5.29.37 Criticality=false > ExtendedKeyUsages [ > clientAuth > serverAuth > ] > > #4: ObjectId: 2.5.29.15 Criticality=true > KeyUsage [ > DigitalSignature > Non_repudiation > Key_Encipherment > Data_Encipherment > Key_Agreement > Key_CertSign > Crl_Sign > ] > > #5: ObjectId: 2.5.29.14 Criticality=false > SubjectKeyIdentifier [ > KeyIdentifier [ > 0000: E1 FC AB 2F 69 4D 6A FE A5 E8 2F B6 43 13 79 1C .../iMj.../.C.y. > 0010: 46 F5 23 91 F.#. > ] > ] > > ******************************************* > ******************************************* > > Output of "nifi-app.log" > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET > /nifi-api/flow/current-user to my.ip.address:443 due to {} > com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: > HTTPS hostname wrong: should be <my.ip.address> > at com.sun.jersey.client.urlconnection.URLConnectionClientHandler. > handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19] > at com.sun.jersey.api.client.Client.handle(Client.java:652) > ~[jersey-client-1.19.jar:1.19] > at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle( > GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.jar:1.19] > at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) > ~[jersey-client-1.19.jar:1.19] > at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) > ~[jersey-client-1.19.jar:1.19] > at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509) > ~[jersey-client-1.19.jar:1.19] > at org.apache.nifi.cluster.coordination.http.replication. > ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578) > ~[nifi-framework-cluster-1.1.1.jar:1.1.1] > at org.apache.nifi.cluster.coordination.http.replication. > ThreadPoolRequestReplicator$NodeHttpRequest.run( > ThreadPoolRequestReplicator.java:770) ~[nifi-framework-cluster-1.1. > 1.jar:1.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_121] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_121] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_121] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_121] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] > Caused by: java.io.IOException: HTTPS hostname wrong: should be > <my.ip.address> > at > sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649) > ~[na:1.8.0_121] > at > sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:573) > ~[na:1.8.0_121] > at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnec > tion.connect(AbstractDelegateHttpsURLConnection.java:185) ~[na:1.8.0_121] > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546) > ~[na:1.8.0_121] > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) > ~[na:1.8.0_121] > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > ~[na:1.8.0_121] > at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode( > HttpsURLConnectionImpl.java:338) ~[na:1.8.0_121] > at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._ > invoke(URLConnectionClientHandler.java:253) ~[jersey-client-1.19.jar:1.19] > at com.sun.jersey.client.urlconnection.URLConnectionClientHandler. > handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19] > ... 12 common frames omitted > > ------------------------------------------------- > > Output of "nifi-user.log" > > 2017-02-28 21:03:47,763 INFO [NiFi Web Server-19] > o.a.n.w.a.c.IllegalStateExceptionMapper > java.lang.IllegalStateException: Kerberos ticket login not supported by > this NiFi.. Returning Conflict response. > 2017-02-28 21:03:47,917 INFO [NiFi Web Server-16] > o.a.n.w.s.NiFiAuthenticationFilter > Attempting request for (CN=admin, OU=NIFI) GET > https://1my.ip.address/nifi-api/flow/current-user (source ip: > my.machine.ip) > 2017-02-28 21:03:47,919 INFO [NiFi Web Server-16] > o.a.n.w.s.NiFiAuthenticationFilter > Authentication success for CN=admin, OU=NIFI > 2017-02-28 21:05:44,793 INFO [NiFi Web Server-22] > o.a.n.w.a.c.IllegalStateExceptionMapper > java.lang.IllegalStateException: Kerberos ticket login not supported by > this NiFi.. Returning Conflict response. > 2017-02-28 21:05:44,892 INFO [NiFi Web Server-20] > o.a.n.w.s.NiFiAuthenticationFilter > Attempting request for (CN=admin, OU=NIFI) GET https://my.ip.address/nifi- > api/flow/current-user (source ip: my.machine.ip) > 2017-02-28 21:05:44,893 INFO [NiFi Web Server-20] > o.a.n.w.s.NiFiAuthenticationFilter > Authentication success for CN=admin, OU=NIFI > > > >
