Hi Ricky, Sorry for the delay in my response. I see a couple things which could be causing an issue:
1. You do not specify a hostname for the NiFi node (nifi.web.https.host=) which you should do. I have not set up a one-node cluster before, but my suspicion is that this field needs to be populated, and the value needs to match the CN for the issued server certificate. This value does not need to be unique (i.e. it could be nifi.cloudera.com <http://nifi.cloudera.com/> or localhost and you could run multiple instances on the same machine, identified by the same certificate and different ports), but I think it has to be populated. 1.a. While you have nifi.security.needClientAuth=false set in both cluster and standalone mode, I am not sure if this allows for the truststore to be missing completely. I would have to explore this further. There is a current Jira for clarifying cluster, UI/API, and S2S TLS configurations [1]. You can try using the default JRE truststore, located at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”. 2. The certificate you have put into the keystore appears to be a certificate identifying you (Ricky the person) rather than a server entity. You should create a certificate identifying the server itself, so the CN is the FQDN as mentioned above. Then import into (or generate directly inside) the keystore. You can also use the provided NiFi TLS Toolkit to automate this entire process [2]. Certificate chain 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzer i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzer compare to an example connection to google.com <http://google.com/>:443: Certificate chain 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com i:/C=US/O=Google Inc/CN=Google Internet Authority G2 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2 i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority 3. It appears the certificate you are using is expired. If you look at the output of the OpenSSL command you ran, you can see that the result code was 10, not 0 as would be in the case of a successful connection (I understand that it looks successful because it negotiates a key and encrypts the channel). New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-SHA384 Session-ID: 581F5A5...41153B Session-ID-ctx: Master-Key: CEEED2...944C30 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1478449748 Timeout : 300 (sec) Verify return code: 10 (certificate has expired) --- HTTP/1.1 400 No URI Content-Length: 0 Connection: close Server: Jetty(9.3.9.v20160517) I rebuilt your certificate from the Base64-encoded version in the output, and it appears it expired on October 25, 2016. You can verify this by copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END CERTIFICATE-----“ (including these markers) into a text file and naming it “ricky.pem” and then running the following command: hw12203:/Users/alopresto/Workspace/scratch (master) alopresto 🔓 11s @ 17:09:30 $ more ricky.pem -----BEGIN CERTIFICATE----- MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD ... 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU= -----END CERTIFICATE----- hw12203:/Users/alopresto/Workspace/scratch (master) alopresto 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 485683207 (0x1cf2f007) Signature Algorithm: sha256WithRSAEncryption Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team, Cloudera, CN=Ricky Saltzer Validity Not Before: Jul 27 17:15:46 2016 GMT Not After : Oct 25 17:15:46 2016 GMT Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team, Cloudera, CN=Ricky Saltzer Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c: 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2: 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a: 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02: c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af: ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7: fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81: ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95: d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d: b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a: 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9: 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c: 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4: 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7: bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22: 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e: 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57: 2b:c5 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 1C:DD:E1:16:5F:41:CC:7C:69:27:E3:B1:6B:78:EF:FA:16:DA:1F:6F Signature Algorithm: sha256WithRSAEncryption da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49: 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62: 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34: 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39: 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9: 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4: c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66: 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24: 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a: 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e: 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3: 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f: f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3: ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0: 72:a0:9f:35 hw12203:/Users/alopresto/Workspace/scratch (master) alopresto 🔓 11s @ 17:09:47 $ I would recommend using the TLS Toolkit to generate a valid CA, server certificate, client certificate, and keystore & truststore (seriously, one command does all of this) and then re-trying. From this stable baseline, I think I’ll be able to better help iron out the cluster/standalone issues. [1] https://issues.apache.org/jira/browse/NIFI-1990 [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 8, 2016, at 1:31 AM, Andre <[email protected]> wrote: > > Rick, > > Can you confirm the certificate has a chain of trust with the default JDK > trusted certs? (i.e. trusted by the JVM) > > Cheers > > > On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[email protected]> wrote: > >> Hey Andy - >> >> Thanks again for the help. >> >> The error message seems indicative that it doesn't seem to properly read >> the keystore file. One thing to note, if I point the nifi properties to a >> bogus keystore location, then it actually throws a FileNotFound exception. >> This is really odd behavior, because as I mentioned I'm able to start it in >> standalone mode using the correct keystore location, just as I try to do in >> clustered mode. >> >> I've attached both the clustered [1] nifi.properties, which doesn't work, >> and the standalone [2] which does work. . I restored it to a more basic >> configuration without the encrypted configuration, but with SSL still >> enabled. I also added a diff [3] of both the standalone and clustered >> properties file. Note that I I only have NiFi configured to use the >> keystore and not a truststore. I've redacted a few of the values in the >> property files, but be assured that the keystore is most definitely valid >> and is readable / locatable, as starting in standalone works just fine. >> >> I ran the SSL command [4] you gave me, minus the three PEM file arguments >> as I don't have any of those on hand. I hope that is fine. The output still >> looks good. >> >> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562 >> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950 >> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558 >> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a >> >> >> >> >> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[email protected]> >> wrote: >> >>> Hi Ricky, >>> >>> Sorry, should have noted that the debug output goes to >> nifi-bootstrap.log, >>> so thanks Mark for jumping in to help there. >>> >>> If you look at the top of that log, you’ll note that there is no keystore >>> file provided and the truststore loaded is the default JRE cacerts >>> truststore. Can you please provide your nifi.properties file in a Gist, >> **taking >>> care to redact any sensitive values** like keystore/truststore passwords, >>> although I think from looking at your log output, you are taking >> advantage >>> of the encrypted configuration feature, so even viewing the encrypted >>> values should be ok. Could you also please provide the directory listing >>> where the keystore and truststore are located including the permissions >> and >>> ownership information? >>> >>> There may be a bug in the logic between cluster and standalone mode, but >> I >>> haven’t encountered this behavior before. If you can start NiFi in >>> standalone mode, could you please provide the output of the following >>> command run from the terminal? It will simulate an HTTPS connection to >> the >>> server and verify the key and certificate presented by NiFi. >>> >>> * host — the NiFi hostname >>> * port — the port NiFi is running on >>> * path_to_your_cert.pem — the public key certificate identifying the >>> client/user (i.e. what you load into your browser to authenticate) >>> * path_to_your_key.key — the private key identifying the client/user >>> * path_to_your_CA_cert.pem — the public key certificate identifying the >> CA >>> which signed your NiFi server certificate (if self-signed, provide that >>> certificate) >>> >>> $ openssl s_client -connect <host:port> -debug -state -cert >>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile >>> <path_to_your_CA_cert.pem> >>> >>> Andy LoPresto >>> [email protected] >>> *[email protected] <[email protected]>* >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[email protected]> wrote: >>> >>> Hey guys - >>> >>> I went ahead and uploaded the boostrap log. I took a look at it and it >>> seems to be the same error [1] >>> >>> [1]: >>> https://gist.githubusercontent.com/rickysaltzer/ >>> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa >>> 9e70c57e49/gistfile1.txt >>> >>> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[email protected]> wrote: >>> >>> Hey Ricky, >>> >>> When you enable debug logging for SSL, it writes to StdErr (or StdOut?) >> so >>> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log. >>> Can you give that a look? >>> >>> Thanks >>> -Mark >>> >>> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[email protected]> wrote: >>> >>> Hey Andy - >>> >>> Thanks for the response. I'm currently just trying to get one node in >>> clustered mode before adding a second. The keystore is stored locally and >>> I've confirmed it's readable, as it was able to start once I took it out >>> >>> of >>> >>> clustered mode. I added that line to the bootstrap.conf, but I don't >>> believe any additional logging was produced in regards to troubleshooting >>> this problem. Just in case, I've attached the entire log [1]. >>> >>> [1]: >>> https://gist.githubusercontent.com/rickysaltzer/ >>> >>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0 >>> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/ >> rickysaltzer/ >>> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0 >>> fedabc88bb/gistfile1.txt> >>> >>> >>> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[email protected] >>> >>> <mailto:[email protected]>> wrote: >>> >>> >>> Hi Ricky, >>> >>> Sorry to hear you are having this issue. Is the keystore available on >>> >>> all >>> >>> nodes of the cluster? It appears from the log message that the keystore >>> >>> is >>> >>> not found during startup. To further debug, you can add the following >>> >>> line >>> >>> in bootstrap.conf to provide additional logging: >>> >>> java.arg.15=-Djavax.net.debug=ssl,handshake >>> >>> Andy LoPresto >>> [email protected] <mailto:[email protected]> >>> *[email protected] <mailto:[email protected]> < >>> >>> [email protected] <mailto:[email protected]>>* >>> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[email protected]> wrote: >>> >>> Hey all - >>> >>> I'm using NiFi 1.0 and I'm having an issue using secure mode with a >>> >>> local >>> >>> key store while in clustered mode. If I set the node in clustered mode, >>> >>> and >>> >>> also provide a valid keystore, I receive a KeyStoreException [1]. If I >>> >>> set >>> >>> the configuration to not use clustered mode, NiFi will start up fine >>> >>> with >>> >>> the provided key store. Am I supposed to be storing this key store in >>> Zookeeper somewhere? >>> >>> >>> [1] >>> >>> >>> Caused by: java.security.KeyStoreException: not found >>> >>> >>> at java.security.KeyStore.getInstance(KeyStore.java:839) >>> ~[na:1.8.0_11] >>> >>> at >>> org.apache.nifi.io.socket.SSLContextFactory.<init>( >>> SSLContextFactory.java:61) >>> ~[nifi-socket-utils-1.0.0.jar:1.0.0] >>> >>> at >>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto >>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45) >>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0] >>> >>> at >>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto >>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30) >>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0] >>> >>> at >>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport. >>> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) >>> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] >>> >>> ... 69 common frames omitted >>> >>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not >>> >>> available >>> >>> >>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159) >>> ~[na:1.8.0_11] >>> >>> at java.security.Security.getImpl(Security.java:695) >>> >>> ~[na:1.8.0_11] >>> >>> >>> at java.security.KeyStore.getInstance(KeyStore.java:836) >>> ~[na:1.8.0_11] >>> >>> ... 73 common frames omitted >>> >>> >>> >>> >>> >>> -- >>> Ricky Saltzer >>> http://www.cloudera.com <http://www.cloudera.com/> >>> >>> >>> >>> >>> >>> -- >>> Ricky Saltzer >>> http://www.cloudera.com >>> >>> >>> >> >> >> -- >> Ricky Saltzer >> http://www.cloudera.com >>
signature.asc
Description: Message signed with OpenPGP using GPGMail
