Dev Team,
I am running into an interesting issue while trying to cluster NiFi in a containerized environment (Docker containers running on DC/OS cluster) and I am somewhat stuck with what to do. I am starting with getting just 2 NiFi nodes with a single external zookeeper instance (just to get it working, will not use for production). Currently our DC/OS cluster does not support container-to-container communication (no overlay network support at the moment) so we are using VIP’s to expose required ports on the container(s) so traffic can be mapped to a well known address and correct container port even though the host/host port may change. - Currently everything spins up and the UI can be accessed on whatever node is elected the Cluster Coordinator (in this case it is the Primary Node as well) (does show that there are 2 nodes in the cluster). However, any action taken on the canvas results in the following error shown in the UI: Node localhost:80 is unable to fulfill this request due to: Transaction c91764e4-2fc8-492b-8887-babb59981ff3 is already in progress. - When trying to access the UI of the other node, the canvas cannot be reached and the following error is shown on the error splash screen (increased read timeout to 30 secs, still the same): An unexpected error has occurred com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out - If configured to use the hostname of the container, then the error is: unknown host exception - In the NiFi logs, the following errors are present (as well as some other warnings): 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to localhost:80 due to com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] o.a.n.c.c.h.r.ThreadPoolRequestReplicator 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at com.sun.jersey.api.client.Client.handle(Client.java:652) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:641) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:852) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.lang.Thread.run(Thread.java:748) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut Caused by: java.net.SocketTimeoutException: Read timed out 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.net.SocketInputStream.socketRead0(Native Method) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.net.SocketInputStream.read(SocketInputStream.java:171) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.net.SocketInputStream.read(SocketInputStream.java:141) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.io.BufferedInputStream.read(BufferedInputStream.java:345) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) 2018-01-29 14:46:52,393 INFO [NiFi logging handler] org.apache.nifi.StdOut ... 12 common frames omitted 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:42,827 WARN [Process Cluster Protocol Request-4] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from <ip-host.aws.internal.address> due to java.io.EOFException 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut java.io.EOFException: null 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:112) 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:142) 2018-01-29 15:19:42,827 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) 2018-01-29 15:19:42,828 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 2018-01-29 15:19:42,828 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 2018-01-29 15:19:42,828 INFO [NiFi logging handler] org.apache.nifi.StdOut at java.lang.Thread.run(Thread.java:748) 2018-01-29 15:19:45,505 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:45,505 INFO [Process Cluster Protocol Request-5] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 181a3bd2-894e-48bf-a8d3-c298d1992524 (type=HEARTBEAT, length=2555 bytes) from localhost:80 in 2 millis 2018-01-29 15:19:46,118 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:46,118 INFO [Process Cluster Protocol Request-6] o.a.n.c.p.impl.SocketProtocolListener Finished processing request f864bf7b-f8fc-40e9-8f04-0cd9d96b3f66 (type=HEARTBEAT, length=2555 bytes) from localhost:80 in 2 millis 2018-01-29 15:19:46,119 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:46,119 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 15:19:46,114 and sent to nifi-node1.my.vip.for.dcos:9001 at 2018-01-29 15:19:46,119; send took 4 millis 2018-01-29 15:19:47,599 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:47,599 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 7490 nanos 2018-01-29 15:19:50,515 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:50,515 INFO [Process Cluster Protocol Request-7] o.a.n.c.p.impl.SocketProtocolListener Finished processing request d9577954-71e3-41eb-b956-c235a98bf6f9 (type=HEARTBEAT, length=2555 bytes) from localhost:80 in 4 millis 2018-01-29 15:19:51,122 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:51,122 INFO [Process Cluster Protocol Request-8] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 18626790-5d19-44ab-af0e-b3246d0a4482 (type=HEARTBEAT, length=2555 bytes) from localhost:80 in 2 millis 2018-01-29 15:19:51,123 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:51,123 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 15:19:51,119 and sent to nifi-node1.my.vip.for.dcos:9001 at 2018-01-29 15:19:51,123; send took 4 millis 2018-01-29 15:19:52,601 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:52,600 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 7776 nanos 2018-01-29 15:19:52,828 INFO [NiFi logging handler] org.apache.nifi.StdOut 2018-01-29 15:19:52,828 WARN [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from <ip-host.aws.internal.address> due to java.io.EOFException 2018-01-29 15:19:52,829 INFO [NiFi logging handler] org.apache.nifi.StdOut java.io.EOFException: null - Here are relevant configs in nifi.properties file: # web properties # nifi.web.war.directory=./lib # for web host, I have tried: localhost, container hostname, left blank (to listen on all interfaces), the VIP configured for external access all of which have failed (read timeout or unknown host exception) nifi.web.http.host= nifi.web.http.port=80 nifi.web.http.network.interface.default= nifi.web.https.host= nifi.web.https.port= nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=nifi-node1.my.vip.for.dcos nifi.cluster.node.protocol.port=9001 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=30 sec nifi.cluster.node.read.timeout=30 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=1 mins nifi.cluster.flow.election.max.candidates= # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=zookeeper.my.vip.for.dcos:2181 nifi.zookeeper.connect.timeout=3 secs nifi.zookeeper.session.timeout=3 secs nifi.zookeeper.root.node=/some-unique-id My suspicion why this is happening is due to the “nifi.web.http.host=” configuration does not have a discoverable address and I am not sure what I can configure this to, to make this work. I have configured a VIP for port 80 so that it can be externalized (and load balanced) and be reached outside of the DCOS cluster via web browser (which works for accessing the UI). It seems that the UI replication relies on the web address of the other nodes (I was thinking that this was all done via the nifi.cluster.node.address and nifi.cluster.node.protocol.port properties, but I may have been wrong. What else is the value of “nifi.web.http.host=” used for? Any ideas or any clarity that can be shed on this would be greatly appreciated! Thanks in Advance, Ryan H.
