Hey Andy - Thanks for the reply, I used the openssl command you provided and indeed the return code was *OK*. Before proceeding with the recommendation of importing the key into my OSX keychain, I would like to understand why this required. When using HTTPS mode in non-clustered mode, it does not require clients to have a special key or cert imported to their machine. Instead, the client is given a warning in the browser, and it's up to them to proceed. This UI will serve as an endpoint to several users, and I would really like to avoid the cumbersomeness of having members of multiple teams follow instructions for importing keys just so they can access a web UI.
On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <[email protected]> wrote: > Hi Ricky, > > The ERR_CONNECTION_CLOSED is likely because you are not sending a client > certificate on the HTTP request. By default, a secured cluster requires > client certificate authentication unless LDAP or Kerberos are configured as > identity providers [1]. The TLS Toolkit provides a quick way to generate a > valid client certificate which you can load into your browser in order to > access the site. > > First, verify the cluster is running and accepting incoming connections > (we’re going to cheat here just to be quick about it; disclaimer that this > is not the RIGHT way to do this): > > In the directory where you ran the toolkit, you noted there was a > “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded > public certificate of the NiFi CA cert that was generated by the toolkit, > and the key file is the PEM-encoded private key. Because this is the same > certificate that signed the NiFi server key loaded in the keystore, it is > also loaded into the truststore. That means the server will accept an > incoming connection with any certificate signed by the CA cert. > Coincidentally, the CA cert is self-signed, so… > > $ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem > -key nifi-key.key -CAfile nifi-cert.pem > > That command will attempt to negotiate a TLS connection to your server by > presenting the CA cert and key as the client. Again, not semantically > correct, but technically will work. You’ll get a long output, but it > should end in a section like this: > > --- > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 > Server public key is 2048 bit > Secure Renegotiation IS supported > Compression: NONE > Expansion: NONE > No ALPN negotiated > SSL-Session: > Protocol : TLSv1.2 > Cipher : ECDHE-RSA-AES256-SHA384 > Session-ID: 583DCD...9E828C > Session-ID-ctx: > Master-Key: 5477C0...A51E85 > Key-Arg : None > PSK identity: None > PSK identity hint: None > SRP username: None > Start Time: 1480445265 > Timeout : 300 (sec) > Verify return code: 0 (ok) > --- > > The important part is the last line — you want the *Verify return code* to > be 0 for success. Once you have verified this, run the TLS toolkit again to > generate a valid client certificate: > > $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi' > -B thisIsABadPassword > > This will generate a PKCS12 keystore (*.p12) containing your public > certificate AND the private key, as well as an additional file (.password) > containing the password you provided for -B. > > hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 > (master) alopresto > 🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, > OU=Apache NiFi' -B password > 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone. > TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using > embedded one. > 2016/11/29 10:50:21 INFO [main] > org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: > Running standalone certificate generation with output directory > ../nifi-toolkit-1.1.0 > 2016/11/29 10:50:21 INFO [main] > org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: > Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key > ../nifi-toolkit-1.1.0/nifi-key.key > 2016/11/29 10:50:21 INFO [main] > org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: > No hostnames specified, not generating any host certificates or > configuration. > 2016/11/29 10:50:21 INFO [main] > org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: > Generating new client certificate ../nifi-toolkit-1.1.0/CN= > Ricky_Saltzer_OU=Apache_NiFi.p12 > 2016/11/29 10:50:21 INFO [main] > org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: > Successfully generated client certificate ../nifi-toolkit-1.1.0/CN= > Ricky_Saltzer_OU=Apache_NiFi.p12 > 2016/11/29 10:50:21 INFO [main] > org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: > tls-toolkit standalone completed successfully > hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 > (master) alopresto > 🔓 7s @ 10:50:22 $ ll > total 136 > drwxr-xr-x 20 alopresto staff 680B Nov 29 10:50 ./ > drwxr-xr-x 4 alopresto staff 136B Nov 28 17:16 ../ > -rw------- 1 alopresto staff 3.4K Nov 29 10:50 > CN=Ricky_Saltzer_OU=Apache_NiFi.p12 > -rw------- 1 alopresto staff 8B Nov 29 10:50 > CN=Ricky_Saltzer_OU=Apache_NiFi.password > ... > -rw------- 1 alopresto staff 1.2K Nov 28 16:31 nifi-cert.pem > -rw------- 1 alopresto staff 1.6K Nov 28 16:31 nifi-key.key > hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 > (master) alopresto > 🔓 4s @ 10:50:27 $ > > You can double-click on the *.p12 file to open it in your default handler > — on OS X for example, this is Keychain Access. You can also manually load > it into Firefox to keep it isolated from your system certificates. > > Now when you visit the UI through the browser, you should receive a prompt > for which certificate to present, and select this entry from the presented > list. > > If you get a NiFi error message in the browser that you do not have > sufficient access, remember to update conf/authorizers.xml with an Initial > Admin Identity, which should match EXACTLY the DN of this certificate — > “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed > authentication message in logs/nifi-user.log to be sure. > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <[email protected]> wrote: > > Hi Andy - > > Thanks again for the help, sorry I've been a little preoccupied with some > other tasks. I took your advice and used the tls-toolkit and things are > looking a lot better. I generated a keystore/truststore for both of my > nodes. It also generated a pem and key file. I was able to get NiFi to > start in clustered mode (horray)! The only issue I'm seeing now is that I > can't actually connect to the web UI using HTTPS, as it immediately throws > a "ERR_CONNECTION_CLOSED" message in my browser. > > Judging by the logs, NiFi is seemingly running. A leader election took > place and the node is heart beating to itself. Am I missing a step where > I'm supposed to be doing something with the pem/key files? > > Thanks again, > Ricky > > On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <[email protected]> > wrote: > > Hi Ricky, > > Sorry for the delay in my response. I see a couple things which could be > causing an issue: > > 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=) > which you should do. I have not set up a one-node cluster before, but my > suspicion is that this field needs to be populated, and the value needs to > match the CN for the issued server certificate. This value does not need to > be unique (i.e. it could be nifi.cloudera.com or localhost and you could > run multiple instances on the same machine, identified by the same > certificate and different ports), but I think it has to be populated. > 1.a. While you have nifi.security.needClientAuth=false set in both > cluster and standalone mode, I am not sure if this allows for the > truststore to be missing completely. I would have to explore this further. > There is a current Jira for clarifying cluster, UI/API, and S2S TLS > configurations [1]. You can try using the default JRE truststore, located > at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”. > 2. The certificate you have put into the keystore appears to be a > certificate identifying you (Ricky the person) rather than a server entity. > You should create a certificate identifying the server itself, so the CN is > the FQDN as mentioned above. Then import into (or generate directly inside) > the keystore. You can also use the provided NiFi TLS Toolkit to automate > this entire process [2]. > Certificate chain > 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky > Saltzer > i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky > Saltzercompare > to an example connection to google.com:443: > > Certificate chain > 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com > <http://google.com>* > i:/C=US/O=Google Inc/CN=Google Internet Authority G2 > 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2 > i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA > 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA > i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority > > 3. It appears the certificate you are using is expired. If you look at the > output of the OpenSSL command you ran, you can see that the result code was > *10*, not *0* as would be in the case of a successful connection (I > understand that it looks successful because it negotiates a key and > encrypts the channel). > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 > Server public key is 2048 bit > Secure Renegotiation IS supported > Compression: NONE > Expansion: NONE > SSL-Session: > Protocol : TLSv1.2 > Cipher : ECDHE-RSA-AES256-SHA384 > Session-ID: 581F5A5...41153B > Session-ID-ctx: > Master-Key: CEEED2...944C30 > Key-Arg : None > PSK identity: None > PSK identity hint: None > SRP username: None > Start Time: 1478449748 > Timeout : 300 (sec) > Verify return code: *10* *(certificate has expired)* > > --- > HTTP/1.1 400 No URI > Content-Length: 0 > Connection: close > Server: Jetty(9.3.9.v20160517) > I rebuilt your certificate from the Base64-encoded version in the output, > and it appears it expired on October 25, 2016. You can verify this by > copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END > CERTIFICATE-----“ (including these markers) into a text file and naming it > “ricky.pem” and then running the following command: > > hw12203:/Users/alopresto/Workspace/scratch (master) alopresto > 🔓 11s @ 17:09:30 $ more ricky.pem > -----BEGIN CERTIFICATE----- > MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD > ... > 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU= > -----END CERTIFICATE----- > hw12203:/Users/alopresto/Workspace/scratch (master) alopresto > 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout > Certificate: > Data: > Version: 3 (0x2) > Serial Number: 485683207 (0x1cf2f007) > Signature Algorithm: sha256WithRSAEncryption > Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team, > Cloudera, CN=Ricky Saltzer > Validity > Not Before: Jul 27 17:15:46 2016 GMT > *Not After : Oct 25 17:15:46 2016 GMT* > > Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team, > Cloudera, CN=Ricky Saltzer > Subject Public Key Info: > Public Key Algorithm: rsaEncryption > Public-Key: (2048 bit) > Modulus: > 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c: > 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2: > 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a: > 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02: > c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af: > ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7: > fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81: > ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95: > d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d: > b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a: > 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9: > 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c: > 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4: > 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7: > bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22: > 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e: > 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57: > 2b:c5 > Exponent: 65537 (0x10001) > X509v3 extensions: > X509v3 Subject Key Identifier: > 1C:DD:E1:16:5F:41:CC:7C:69:27: > E3:B1:6B:78:EF:FA:16:DA:1F:6F > Signature Algorithm: sha256WithRSAEncryption > da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49: > 30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62: > 0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34: > 1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39: > 05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9: > 25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4: > c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66: > 94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24: > 5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a: > 4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e: > 76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3: > 5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f: > f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3: > ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0: > 72:a0:9f:35 > hw12203:/Users/alopresto/Workspace/scratch (master) alopresto > 🔓 11s @ 17:09:47 $ > > I would recommend using the TLS Toolkit to generate a valid CA, server > certificate, client certificate, and keystore & truststore (seriously, one > command does all of this) and then re-trying. From this stable baseline, I > think I’ll be able to better help iron out the cluster/standalone issues. > > [1] https://issues.apache.org/jira/browse/NIFI-1990 > [2] https://nifi.apache.org/docs/nifi-docs/html/ > administration-guide.html#tls-generation-toolkit > > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 8, 2016, at 1:31 AM, Andre <[email protected]> wrote: > > Rick, > > Can you confirm the certificate has a chain of trust with the default JDK > trusted certs? (i.e. trusted by the JVM) > > Cheers > > > On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[email protected]> wrote: > > Hey Andy - > > Thanks again for the help. > > The error message seems indicative that it doesn't seem to properly read > the keystore file. One thing to note, if I point the nifi properties to a > bogus keystore location, then it actually throws a FileNotFound exception. > This is really odd behavior, because as I mentioned I'm able to start it in > standalone mode using the correct keystore location, just as I try to do in > clustered mode. > > I've attached both the clustered [1] nifi.properties, which doesn't work, > and the standalone [2] which does work. . I restored it to a more basic > configuration without the encrypted configuration, but with SSL still > enabled. I also added a diff [3] of both the standalone and clustered > properties file. Note that I I only have NiFi configured to use the > keystore and not a truststore. I've redacted a few of the values in the > property files, but be assured that the keystore is most definitely valid > and is readable / locatable, as starting in standalone works just fine. > > I ran the SSL command [4] you gave me, minus the three PEM file arguments > as I don't have any of those on hand. I hope that is fine. The output still > looks good. > > [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562 > [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950 > [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558 > [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a > > > > > On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[email protected]> > wrote: > > Hi Ricky, > > Sorry, should have noted that the debug output goes to > > nifi-bootstrap.log, > > so thanks Mark for jumping in to help there. > > If you look at the top of that log, you’ll note that there is no keystore > file provided and the truststore loaded is the default JRE cacerts > truststore. Can you please provide your nifi.properties file in a Gist, > > **taking > > care to redact any sensitive values** like keystore/truststore passwords, > although I think from looking at your log output, you are taking > > advantage > > of the encrypted configuration feature, so even viewing the encrypted > values should be ok. Could you also please provide the directory listing > where the keystore and truststore are located including the permissions > > and > > ownership information? > > There may be a bug in the logic between cluster and standalone mode, but > > I > > haven’t encountered this behavior before. If you can start NiFi in > standalone mode, could you please provide the output of the following > command run from the terminal? It will simulate an HTTPS connection to > > the > > server and verify the key and certificate presented by NiFi. > > * host — the NiFi hostname > * port — the port NiFi is running on > * path_to_your_cert.pem — the public key certificate identifying the > client/user (i.e. what you load into your browser to authenticate) > * path_to_your_key.key — the private key identifying the client/user > * path_to_your_CA_cert.pem — the public key certificate identifying the > > CA > > which signed your NiFi server certificate (if self-signed, provide that > certificate) > > $ openssl s_client -connect <host:port> -debug -state -cert > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile > <path_to_your_CA_cert.pem> > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[email protected]> wrote: > > Hey guys - > > I went ahead and uploaded the boostrap log. I took a look at it and it > seems to be the same error [1] > > [1]: > https://gist.githubusercontent.com/rickysaltzer/ > b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa > 9e70c57e49/gistfile1.txt > > On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[email protected]> wrote: > > Hey Ricky, > > When you enable debug logging for SSL, it writes to StdErr (or StdOut?) > > so > > it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log. > Can you give that a look? > > Thanks > -Mark > > On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[email protected]> wrote: > > Hey Andy - > > Thanks for the response. I'm currently just trying to get one node in > clustered mode before adding a second. The keystore is stored locally and > I've confirmed it's readable, as it was able to start once I took it out > > of > > clustered mode. I added that line to the bootstrap.conf, but I don't > believe any additional logging was produced in regards to troubleshooting > this problem. Just in case, I've attached the entire log [1]. > > [1]: > https://gist.githubusercontent.com/rickysaltzer/ > > ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0 > fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/ > > rickysaltzer/ > > ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0 > fedabc88bb/gistfile1.txt> > > > On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[email protected] > > <mailto:[email protected] <[email protected]> <[email protected]>>> > wrote: > > > Hi Ricky, > > Sorry to hear you are having this issue. Is the keystore available on > > all > > nodes of the cluster? It appears from the log message that the keystore > > is > > not found during startup. To further debug, you can add the following > > line > > in bootstrap.conf to provide additional logging: > > java.arg.15=-Djavax.net.debug=ssl,handshake > > Andy LoPresto > [email protected] <mailto:[email protected] <[email protected]> < > [email protected]>> > *[email protected] <mailto:[email protected] > <[email protected]> > <[email protected]>> < > > [email protected] <mailto:[email protected] > <[email protected]> > > <[email protected]>>>* > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[email protected]> wrote: > > Hey all - > > I'm using NiFi 1.0 and I'm having an issue using secure mode with a > > local > > key store while in clustered mode. If I set the node in clustered mode, > > and > > also provide a valid keystore, I receive a KeyStoreException [1]. If I > > set > > the configuration to not use clustered mode, NiFi will start up fine > > with > > the provided key store. Am I supposed to be storing this key store in > Zookeeper somewhere? > > > [1] > > > Caused by: java.security.KeyStoreException: not found > > > at java.security.KeyStore.getInstance(KeyStore.java:839) > ~[na:1.8.0_11] > > at > org.apache.nifi.io.socket.SSLContextFactory.<init>( > SSLContextFactory.java:61) > ~[nifi-socket-utils-1.0.0.jar:1.0.0] > > at > org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto > ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45) > ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0] > > at > org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto > ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30) > ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0] > > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport. > doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) > ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] > > ... 69 common frames omitted > > Caused by: java.security.NoSuchAlgorithmException: KeyStore not > > available > > > at sun.security.jca.GetInstance.getInstance(GetInstance.java:159) > ~[na:1.8.0_11] > > at java.security.Security.getImpl(Security.java:695) > > ~[na:1.8.0_11] > > > at java.security.KeyStore.getInstance(KeyStore.java:836) > ~[na:1.8.0_11] > > ... 73 common frames omitted > > > > > > -- > Ricky Saltzer > http://www.cloudera.com <http://www.cloudera.com/> > > > > > > -- > Ricky Saltzer > http://www.cloudera.com > > > > > > -- > Ricky Saltzer > http://www.cloudera.com > > > > > > -- > Ricky Saltzer > http://www.cloudera.com > > > -- Ricky Saltzer http://www.cloudera.com
