Hey Andy - Thanks again for the help.
The error message seems indicative that it doesn't seem to properly read the keystore file. One thing to note, if I point the nifi properties to a bogus keystore location, then it actually throws a FileNotFound exception. This is really odd behavior, because as I mentioned I'm able to start it in standalone mode using the correct keystore location, just as I try to do in clustered mode. I've attached both the clustered [1] nifi.properties, which doesn't work, and the standalone [2] which does work. . I restored it to a more basic configuration without the encrypted configuration, but with SSL still enabled. I also added a diff [3] of both the standalone and clustered properties file. Note that I I only have NiFi configured to use the keystore and not a truststore. I've redacted a few of the values in the property files, but be assured that the keystore is most definitely valid and is readable / locatable, as starting in standalone works just fine. I ran the SSL command [4] you gave me, minus the three PEM file arguments as I don't have any of those on hand. I hope that is fine. The output still looks good. [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562 [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950 [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558 [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[email protected]> wrote: > Hi Ricky, > > Sorry, should have noted that the debug output goes to nifi-bootstrap.log, > so thanks Mark for jumping in to help there. > > If you look at the top of that log, you’ll note that there is no keystore > file provided and the truststore loaded is the default JRE cacerts > truststore. Can you please provide your nifi.properties file in a Gist, > **taking > care to redact any sensitive values** like keystore/truststore passwords, > although I think from looking at your log output, you are taking advantage > of the encrypted configuration feature, so even viewing the encrypted > values should be ok. Could you also please provide the directory listing > where the keystore and truststore are located including the permissions and > ownership information? > > There may be a bug in the logic between cluster and standalone mode, but I > haven’t encountered this behavior before. If you can start NiFi in > standalone mode, could you please provide the output of the following > command run from the terminal? It will simulate an HTTPS connection to the > server and verify the key and certificate presented by NiFi. > > * host — the NiFi hostname > * port — the port NiFi is running on > * path_to_your_cert.pem — the public key certificate identifying the > client/user (i.e. what you load into your browser to authenticate) > * path_to_your_key.key — the private key identifying the client/user > * path_to_your_CA_cert.pem — the public key certificate identifying the CA > which signed your NiFi server certificate (if self-signed, provide that > certificate) > > $ openssl s_client -connect <host:port> -debug -state -cert > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile > <path_to_your_CA_cert.pem> > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[email protected]> wrote: > > Hey guys - > > I went ahead and uploaded the boostrap log. I took a look at it and it > seems to be the same error [1] > > [1]: > https://gist.githubusercontent.com/rickysaltzer/ > b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa > 9e70c57e49/gistfile1.txt > > On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[email protected]> wrote: > > Hey Ricky, > > When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so > it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log. > Can you give that a look? > > Thanks > -Mark > > On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[email protected]> wrote: > > Hey Andy - > > Thanks for the response. I'm currently just trying to get one node in > clustered mode before adding a second. The keystore is stored locally and > I've confirmed it's readable, as it was able to start once I took it out > > of > > clustered mode. I added that line to the bootstrap.conf, but I don't > believe any additional logging was produced in regards to troubleshooting > this problem. Just in case, I've attached the entire log [1]. > > [1]: > https://gist.githubusercontent.com/rickysaltzer/ > > ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0 > fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/ > ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0 > fedabc88bb/gistfile1.txt> > > > On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[email protected] > > <mailto:[email protected]>> wrote: > > > Hi Ricky, > > Sorry to hear you are having this issue. Is the keystore available on > > all > > nodes of the cluster? It appears from the log message that the keystore > > is > > not found during startup. To further debug, you can add the following > > line > > in bootstrap.conf to provide additional logging: > > java.arg.15=-Djavax.net.debug=ssl,handshake > > Andy LoPresto > [email protected] <mailto:[email protected]> > *[email protected] <mailto:[email protected]> < > > [email protected] <mailto:[email protected]>>* > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[email protected]> wrote: > > Hey all - > > I'm using NiFi 1.0 and I'm having an issue using secure mode with a > > local > > key store while in clustered mode. If I set the node in clustered mode, > > and > > also provide a valid keystore, I receive a KeyStoreException [1]. If I > > set > > the configuration to not use clustered mode, NiFi will start up fine > > with > > the provided key store. Am I supposed to be storing this key store in > Zookeeper somewhere? > > > [1] > > > Caused by: java.security.KeyStoreException: not found > > > at java.security.KeyStore.getInstance(KeyStore.java:839) > ~[na:1.8.0_11] > > at > org.apache.nifi.io.socket.SSLContextFactory.<init>( > SSLContextFactory.java:61) > ~[nifi-socket-utils-1.0.0.jar:1.0.0] > > at > org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto > ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45) > ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0] > > at > org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto > ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30) > ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0] > > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport. > doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) > ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] > > ... 69 common frames omitted > > Caused by: java.security.NoSuchAlgorithmException: KeyStore not > > available > > > at sun.security.jca.GetInstance.getInstance(GetInstance.java:159) > ~[na:1.8.0_11] > > at java.security.Security.getImpl(Security.java:695) > > ~[na:1.8.0_11] > > > at java.security.KeyStore.getInstance(KeyStore.java:836) > ~[na:1.8.0_11] > > ... 73 common frames omitted > > > > > > -- > Ricky Saltzer > http://www.cloudera.com <http://www.cloudera.com/> > > > > > > -- > Ricky Saltzer > http://www.cloudera.com > > > -- Ricky Saltzer http://www.cloudera.com
