Hi All,
I am working on setting up a 2 node secure NiFi Cluster (version 1.1.1). I
am using the NiFi Toolkit in standalone mode for the cert and other
information. The 2 nodes are on AWS EC2 instances (1 node per EC2
instance). I am using and embedded Zookeeper for each instance for now, and
will move to use external Zookeeper (3 instances) once I have worked out
the following setup.
There do not look to be any errors during startup. As soon as making the
request from my web browser (with the client cert created from the
Toolkit), It results in an error. After that all looks well in the logs
with regard to recurring errors.
The error displayed in the UI is: "An unexpected error has occurred, please
check the logs"
Searching thru the logs I am seeing the following errors:
*nifi-app.log*
2017-03-16 12:58:44,199 WARN [Replicate Request Thread-5]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
/nifi-api/flow/current-user to my-server-hostname:443 due to {}
com.sun.jersey.api.client.ClientHandlerException:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during
handshake
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
~[jersey-client-1.19.jar:1.19]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
~[nifi-framework-cluster-1.1.1.jar:1.1.1]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
~[nifi-framework-cluster-1.1.1.jar:1.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed
connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
~[na:1.8.0_121]
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
~[na:1.8.0_121]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
~[na:1.8.0_121]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
~[na:1.8.0_121]
at
sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
~[na:1.8.0_121]
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
~[na:1.8.0_121]
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
~[na:1.8.0_121]
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
~[na:1.8.0_121]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
~[na:1.8.0_121]
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
~[na:1.8.0_121]
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
~[jersey-client-1.19.jar:1.19]
... 12 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[na:1.8.0_121]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
~[na:1.8.0_121]
... 23 common frames omitted
2017-03-16 12:58:44,200 WARN [Replicate Request Thread-5]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator
com.sun.jersey.api.client.ClientHandlerException:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during
handshake
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
~[jersey-client-1.19.jar:1.19]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
~[nifi-framework-cluster-1.1.1.jar:1.1.1]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
~[nifi-framework-cluster-1.1.1.jar:1.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed
connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
~[na:1.8.0_121]
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
~[na:1.8.0_121]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
~[na:1.8.0_121]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
~[na:1.8.0_121]
at
sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
~[na:1.8.0_121]
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
~[na:1.8.0_121]
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
~[na:1.8.0_121]
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
~[na:1.8.0_121]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
~[na:1.8.0_121]
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
~[na:1.8.0_121]
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
~[jersey-client-1.19.jar:1.19]
... 12 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[na:1.8.0_121]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
~[na:1.8.0_121]
... 23 common frames omitted
*nifi-bootstrap.log (additional logging turned on in bootstrap.config)*
2017-03-16 12:39:41,195 INFO [NiFi logging handler] org.apache.nifi.StdOut
*** ServerHelloDone
2017-03-16 12:39:41,195 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-117, WRITE: TLSv1.2 Handshake, length = 2225
2017-03-16 12:39:41,196 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-16, WRITE: TLSv1.2 Application Data, length = 16384
2017-03-16 12:39:41,197 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-16, WRITE: TLSv1.2 Application Data, length = 5589
2017-03-16 12:39:41,197 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-16, WRITE: TLSv1.2 Application Data, length = 7213
2017-03-16 12:39:41,197 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-16, WRITE: TLSv1.2 Application Data, length = 7
2017-03-16 12:39:41,270 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-122, WRITE: TLSv1.2 Application Data, length = 16384
2017-03-16 12:39:41,270 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-122, WRITE: TLSv1.2 Application Data, length = 5378
2017-03-16 12:39:41,272 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-122, WRITE: TLSv1.2 Application Data, length = 16384
2017-03-16 12:39:41,275 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-118, called closeInbound()
*2017-03-16 12:39:41,275 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-118, fatal error: 80: Inbound closed before receiving
peer's close_notify: possible truncation attack?*
*2017-03-16 12:39:41,275 INFO [NiFi logging handler] org.apache.nifi.StdOut
javax.net.ssl.SSLException: Inbound closed before receiving peer's
close_notify: possible truncation attack?*
2017-03-16 12:39:41,275 INFO [NiFi logging handler] org.apache.nifi.StdOut
%% Invalidated: [Session-7405, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
2017-03-16 12:39:41,275 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-118, SEND TLSv1.2 ALERT: fatal, description =
internal_error
2017-03-16 12:39:41,275 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-118, WRITE: TLSv1.2 Alert, length = 2
2017-03-16 12:39:41,282 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-117, WRITE: TLSv1.2 Application Data, length = 7663
2017-03-16 12:39:41,282 INFO [NiFi logging handler] org.apache.nifi.StdOut
NiFi Web Server-117, WRITE: TLSv1.2 Application Data, length = 7
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Allow unsafe renegotiation: false
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Allow legacy hello messages: true
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Is initial handshake: true
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Is secure renegotiation: false
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Replicate Request Thread-4, setSoTimeout(5000) called
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring disabled protocol: SSLv3
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
for TLSv1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
for TLSv1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
for TLSv1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for
TLSv1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for
TLSv1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for
TLSv1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
for TLSv1.1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
for TLSv1.1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for
TLSv1.1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
for TLSv1.1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for
TLSv1.1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for
TLSv1.1
2017-03-16 12:39:41,289 INFO [NiFi logging handler] org.apache.nifi.StdOut
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for
TLSv1.1
2017-03-16 12:39:41,290 INFO [NiFi logging handler] org.apache.nifi.StdOut
%% No cached client session
2017-03-16 12:39:41,290 INFO [NiFi logging handler] org.apache.nifi.StdOut
*** ClientHello, TLSv1.2
*nifi-user.log*
2017-03-16 12:39:41,110 INFO [NiFi Web Server-117]
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
Kerberos ticket login not supported by this NiFi.. Returning Conflict
response.
2017-03-16 12:39:41,280 INFO [NiFi Web Server-18]
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=adminuser,
OU=NIFI) GET https://my-server-ip-address/nifi-api/flow/current-user
(source ip: my-local-ip-address)
2017-03-16 12:39:41,280 INFO [NiFi Web Server-18]
o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=adminuser,
OU=NIFI
Main articles followed for the secure cluster setup and for points of
reference:
http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security-configuration
Any help or direction is greatly appreciated!
Cheers,
Ryan H.
