Ryan, I'm not that familiar with Docker and DCOS, but I think what you said is correct...
The issue is that you currently can't leave nifi.web.http.host blank because that will cause the "node API address" of each node to be calculated as 'localhost', which then means replication of requests fails. So you have to set nifi.web.http.host to something that each node can reach. I'm not familiar enough with your setup to know if there is a way to do that. -Bryan On Mon, Jan 29, 2018 at 12:39 PM, Ryan H <[email protected]> wrote: > Hi Bryan, > > Yes that makes total sense, and it is what I figured was happening. So > whatever is configured for nifi.web.http.host is where api calls will go to, > but this is also what jetty will bind to, correct? So in my case, I would > have to have the additional property mentioned in > https://issues.apache.org/jira/browse/NIFI-3642 since jetty can't bind to a > VIP. > > -Ryan H > > On Mon, Jan 29, 2018 at 12:28 PM, Bryan Bende <[email protected]> wrote: >> >> Ryan, >> >> I remember creating an issue for something that seems similar to what >> you are running into: >> >> https://issues.apache.org/jira/browse/NIFI-3642 >> >> Long story short, I believe you do need to specify a value for >> nifi.web.http.host because that will be used to replicate requests >> that come in to the REST API, so each node needs that value to be >> something that is reachable by the other nodes. >> >> -Bryan >> >> >> On Mon, Jan 29, 2018 at 12:03 PM, Ryan H >> <[email protected]> wrote: >> > Dev Team, >> > >> > >> > I am running into an interesting issue while trying to cluster NiFi in a >> > containerized environment (Docker containers running on DC/OS cluster) >> > and I >> > am somewhat stuck with what to do. I am starting with getting just 2 >> > NiFi >> > nodes with a single external zookeeper instance (just to get it working, >> > will not use for production). Currently our DC/OS cluster does not >> > support >> > container-to-container communication (no overlay network support at the >> > moment) so we are using VIP’s to expose required ports on the >> > container(s) >> > so traffic can be mapped to a well known address and correct container >> > port >> > even though the host/host port may change. >> > >> > >> > Currently everything spins up and the UI can be accessed on whatever >> > node is >> > elected the Cluster Coordinator (in this case it is the Primary Node as >> > well) (does show that there are 2 nodes in the cluster). However, any >> > action >> > taken on the canvas results in the following error shown in the UI: >> > >> > >> > Node localhost:80 is unable to fulfill this request due to: Transaction >> > c91764e4-2fc8-492b-8887-babb59981ff3 is already in progress. >> > >> > >> > When trying to access the UI of the other node, the canvas cannot be >> > reached >> > and the following error is shown on the error splash screen (increased >> > read >> > timeout to 30 secs, still the same): >> > >> > >> > An unexpected error has occurred >> > com.sun.jersey.api.client.ClientHandlerException: >> > java.net.SocketTimeoutException: Read timed out >> > >> > >> > If configured to use the hostname of the container, then the error is: >> > >> > unknown host exception >> > >> > >> > In the NiFi logs, the following errors are present (as well as some >> > other >> > warnings): >> > >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request >> > GET >> > /nifi-api/flow/current-user to localhost:80 due to >> > com.sun.jersey.api.client.ClientHandlerException: >> > java.net.SocketTimeoutException: Read timed out >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > com.sun.jersey.api.client.ClientHandlerException: >> > java.net.SocketTimeoutException: Read timed out >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at com.sun.jersey.api.client.Client.handle(Client.java:652) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:641) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:852) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.lang.Thread.run(Thread.java:748) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > Caused by: java.net.SocketTimeoutException: Read timed out >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.net.SocketInputStream.socketRead0(Native Method) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.net.SocketInputStream.read(SocketInputStream.java:171) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.net.SocketInputStream.read(SocketInputStream.java:141) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) >> > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > ... 12 common frames omitted >> > >> > >> > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:42,827 WARN [Process Cluster Protocol Request-4] >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message >> > from <ip-host.aws.internal.address> due to java.io.EOFException >> > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > java.io.EOFException: null >> > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:112) >> > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:142) >> > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) >> > >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > at java.lang.Thread.run(Thread.java:748) >> > >> > 2018-01-29 15:19:45,505 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:45,505 INFO [Process Cluster Protocol Request-5] >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >> > 181a3bd2-894e-48bf-a8d3-c298d1992524 (type=HEARTBEAT, length=2555 bytes) >> > from localhost:80 in 2 millis >> > >> > 2018-01-29 15:19:46,118 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:46,118 INFO [Process Cluster Protocol Request-6] >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >> > f864bf7b-f8fc-40e9-8f04-0cd9d96b3f66 (type=HEARTBEAT, length=2555 bytes) >> > from localhost:80 in 2 millis >> > >> > 2018-01-29 15:19:46,119 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:46,119 INFO [Clustering Tasks Thread-1] >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 >> > 15:19:46,114 and sent to nifi-node1.my.vip.for.dcos:9001 at 2018-01-29 >> > 15:19:46,119; send took 4 millis >> > >> > 2018-01-29 15:19:47,599 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:47,599 INFO [Heartbeat Monitor Thread-1] >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in >> > 7490 nanos >> > >> > 2018-01-29 15:19:50,515 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:50,515 INFO [Process Cluster Protocol Request-7] >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >> > d9577954-71e3-41eb-b956-c235a98bf6f9 (type=HEARTBEAT, length=2555 bytes) >> > from localhost:80 in 4 millis >> > >> > 2018-01-29 15:19:51,122 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:51,122 INFO [Process Cluster Protocol Request-8] >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >> > 18626790-5d19-44ab-af0e-b3246d0a4482 (type=HEARTBEAT, length=2555 bytes) >> > from localhost:80 in 2 millis >> > >> > 2018-01-29 15:19:51,123 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:51,123 INFO [Clustering Tasks Thread-1] >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 >> > 15:19:51,119 and sent to nifi-node1.my.vip.for.dcos:9001 at 2018-01-29 >> > 15:19:51,123; send took 4 millis >> > >> > 2018-01-29 15:19:52,601 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:52,600 INFO [Heartbeat Monitor Thread-1] >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in >> > 7776 nanos >> > >> > 2018-01-29 15:19:52,828 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > 2018-01-29 15:19:52,828 WARN [Process Cluster Protocol Request-9] >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message >> > from <ip-host.aws.internal.address> due to java.io.EOFException >> > >> > 2018-01-29 15:19:52,829 INFO [NiFi logging handler] >> > org.apache.nifi.StdOut >> > java.io.EOFException: null >> > >> > >> > >> > Here are relevant configs in nifi.properties file: >> > >> > # web properties # >> > >> > nifi.web.war.directory=./lib >> > >> > # for web host, I have tried: localhost, container hostname, left blank >> > (to >> > listen on all interfaces), the VIP configured for external access all of >> > which have failed (read timeout or unknown host exception) >> > >> > nifi.web.http.host= >> > >> > nifi.web.http.port=80 >> > >> > nifi.web.http.network.interface.default= >> > >> > nifi.web.https.host= >> > >> > nifi.web.https.port= >> > >> > nifi.web.https.network.interface.default= >> > >> > nifi.web.jetty.working.directory=./work/jetty >> > >> > nifi.web.jetty.threads=200 >> > >> > >> > # cluster node properties (only configure for cluster nodes) # >> > >> > nifi.cluster.is.node=true >> > >> > nifi.cluster.node.address=nifi-node1.my.vip.for.dcos >> > >> > nifi.cluster.node.protocol.port=9001 >> > >> > nifi.cluster.node.protocol.threads=10 >> > >> > nifi.cluster.node.protocol.max.threads=50 >> > >> > nifi.cluster.node.event.history.size=25 >> > >> > nifi.cluster.node.connection.timeout=30 sec >> > >> > nifi.cluster.node.read.timeout=30 sec >> > >> > nifi.cluster.node.max.concurrent.requests=100 >> > >> > nifi.cluster.firewall.file= >> > >> > nifi.cluster.flow.election.max.wait.time=1 mins >> > >> > nifi.cluster.flow.election.max.candidates= >> > >> > >> > # zookeeper properties, used for cluster management # >> > >> > nifi.zookeeper.connect.string=zookeeper.my.vip.for.dcos:2181 >> > >> > nifi.zookeeper.connect.timeout=3 secs >> > >> > nifi.zookeeper.session.timeout=3 secs >> > >> > nifi.zookeeper.root.node=/some-unique-id >> > >> > >> > >> > My suspicion why this is happening is due to the “nifi.web.http.host=” >> > configuration does not have a discoverable address and I am not sure >> > what I >> > can configure this to, to make this work. I have configured a VIP for >> > port >> > 80 so that it can be externalized (and load balanced) and be reached >> > outside >> > of the DCOS cluster via web browser (which works for accessing the UI). >> > It >> > seems that the UI replication relies on the web address of the other >> > nodes >> > (I was thinking that this was all done via the nifi.cluster.node.address >> > and >> > nifi.cluster.node.protocol.port properties, but I may have been wrong. >> > What >> > else is the value of “nifi.web.http.host=” used for? >> > >> > >> > Any ideas or any clarity that can be shed on this would be greatly >> > appreciated! >> > >> > >> > >> > Thanks in Advance, >> > >> > >> > Ryan H. >> > >> > > >
