Hi Bryan, Yes that makes total sense, and it is what I figured was happening. So whatever is configured for nifi.web.http.host is where api calls will go to, but this is also what jetty will bind to, correct? So in my case, I would have to have the additional property mentioned in https://issues.apache.org/jira/browse/NIFI-3642 since jetty can't bind to a VIP.
-Ryan H On Mon, Jan 29, 2018 at 12:28 PM, Bryan Bende <[email protected]> wrote: > Ryan, > > I remember creating an issue for something that seems similar to what > you are running into: > > https://issues.apache.org/jira/browse/NIFI-3642 > > Long story short, I believe you do need to specify a value for > nifi.web.http.host because that will be used to replicate requests > that come in to the REST API, so each node needs that value to be > something that is reachable by the other nodes. > > -Bryan > > > On Mon, Jan 29, 2018 at 12:03 PM, Ryan H > <[email protected]> wrote: > > Dev Team, > > > > > > I am running into an interesting issue while trying to cluster NiFi in a > > containerized environment (Docker containers running on DC/OS cluster) > and I > > am somewhat stuck with what to do. I am starting with getting just 2 NiFi > > nodes with a single external zookeeper instance (just to get it working, > > will not use for production). Currently our DC/OS cluster does not > support > > container-to-container communication (no overlay network support at the > > moment) so we are using VIP’s to expose required ports on the > container(s) > > so traffic can be mapped to a well known address and correct container > port > > even though the host/host port may change. > > > > > > Currently everything spins up and the UI can be accessed on whatever > node is > > elected the Cluster Coordinator (in this case it is the Primary Node as > > well) (does show that there are 2 nodes in the cluster). However, any > action > > taken on the canvas results in the following error shown in the UI: > > > > > > Node localhost:80 is unable to fulfill this request due to: Transaction > > c91764e4-2fc8-492b-8887-babb59981ff3 is already in progress. > > > > > > When trying to access the UI of the other node, the canvas cannot be > reached > > and the following error is shown on the error splash screen (increased > read > > timeout to 30 secs, still the same): > > > > > > An unexpected error has occurred > > com.sun.jersey.api.client.ClientHandlerException: > > java.net.SocketTimeoutException: Read timed out > > > > > > If configured to use the hostname of the container, then the error is: > > > > unknown host exception > > > > > > In the NiFi logs, the following errors are present (as well as some other > > warnings): > > > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request > GET > > /nifi-api/flow/current-user to localhost:80 due to > > com.sun.jersey.api.client.ClientHandlerException: > > java.net.SocketTimeoutException: Read timed out > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > com.sun.jersey.api.client.ClientHandlerException: > > java.net.SocketTimeoutException: Read timed out > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle( > URLConnectionClientHandler.java:155) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at com.sun.jersey.api.client.Client.handle(Client.java:652) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle( > GZIPContentEncodingFilter.java:123) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at com.sun.jersey.api.client.WebResource$Builder.get( > WebResource.java:509) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > org.apache.nifi.cluster.coordination.http.replication. > ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator. > java:641) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > org.apache.nifi.cluster.coordination.http.replication. > ThreadPoolRequestReplicator$NodeHttpRequest.run( > ThreadPoolRequestReplicator.java:852) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.lang.Thread.run(Thread.java:748) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > Caused by: java.net.SocketTimeoutException: Read timed out > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.net.SocketInputStream.read(SocketInputStream.java:171) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.net.SocketInputStream.read(SocketInputStream.java:141) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > sun.net.www.protocol.http.HttpURLConnection.getInputStream0( > HttpURLConnection.java:1587) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > sun.net.www.protocol.http.HttpURLConnection.getInputStream( > HttpURLConnection.java:1492) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.net.HttpURLConnection.getResponseCode( > HttpURLConnection.java:480) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke( > URLConnectionClientHandler.java:253) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle( > URLConnectionClientHandler.java:153) > > > > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > ... 12 common frames omitted > > > > > > > > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:42,827 WARN [Process Cluster Protocol Request-4] > > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message > > from <ip-host.aws.internal.address> due to java.io.EOFException > > > > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > java.io.EOFException: null > > > > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2. > unmarshal(JaxbProtocolContext.java:112) > > > > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > org.apache.nifi.cluster.protocol.impl.SocketProtocolListener. > dispatchRequest(SocketProtocolListener.java:142) > > > > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at org.apache.nifi.io.socket.SocketListener$2$1.run( > SocketListener.java:136) > > > > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > > > > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > > > > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > at java.lang.Thread.run(Thread.java:748) > > > > 2018-01-29 15:19:45,505 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:45,505 INFO [Process Cluster Protocol Request-5] > > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > > 181a3bd2-894e-48bf-a8d3-c298d1992524 (type=HEARTBEAT, length=2555 bytes) > > from localhost:80 in 2 millis > > > > 2018-01-29 15:19:46,118 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:46,118 INFO [Process Cluster Protocol Request-6] > > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > > f864bf7b-f8fc-40e9-8f04-0cd9d96b3f66 (type=HEARTBEAT, length=2555 bytes) > > from localhost:80 in 2 millis > > > > 2018-01-29 15:19:46,119 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:46,119 INFO [Clustering Tasks Thread-1] > > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 > > 15:19:46,114 and sent to nifi-node1.my.vip.for.dcos:9001 at 2018-01-29 > > 15:19:46,119; send took 4 millis > > > > 2018-01-29 15:19:47,599 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:47,599 INFO [Heartbeat Monitor Thread-1] > > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in > > 7490 nanos > > > > 2018-01-29 15:19:50,515 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:50,515 INFO [Process Cluster Protocol Request-7] > > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > > d9577954-71e3-41eb-b956-c235a98bf6f9 (type=HEARTBEAT, length=2555 bytes) > > from localhost:80 in 4 millis > > > > 2018-01-29 15:19:51,122 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:51,122 INFO [Process Cluster Protocol Request-8] > > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > > 18626790-5d19-44ab-af0e-b3246d0a4482 (type=HEARTBEAT, length=2555 bytes) > > from localhost:80 in 2 millis > > > > 2018-01-29 15:19:51,123 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:51,123 INFO [Clustering Tasks Thread-1] > > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 > > 15:19:51,119 and sent to nifi-node1.my.vip.for.dcos:9001 at 2018-01-29 > > 15:19:51,123; send took 4 millis > > > > 2018-01-29 15:19:52,601 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:52,600 INFO [Heartbeat Monitor Thread-1] > > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in > > 7776 nanos > > > > 2018-01-29 15:19:52,828 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > 2018-01-29 15:19:52,828 WARN [Process Cluster Protocol Request-9] > > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message > > from <ip-host.aws.internal.address> due to java.io.EOFException > > > > 2018-01-29 15:19:52,829 INFO [NiFi logging handler] > org.apache.nifi.StdOut > > java.io.EOFException: null > > > > > > > > Here are relevant configs in nifi.properties file: > > > > # web properties # > > > > nifi.web.war.directory=./lib > > > > # for web host, I have tried: localhost, container hostname, left blank > (to > > listen on all interfaces), the VIP configured for external access all of > > which have failed (read timeout or unknown host exception) > > > > nifi.web.http.host= > > > > nifi.web.http.port=80 > > > > nifi.web.http.network.interface.default= > > > > nifi.web.https.host= > > > > nifi.web.https.port= > > > > nifi.web.https.network.interface.default= > > > > nifi.web.jetty.working.directory=./work/jetty > > > > nifi.web.jetty.threads=200 > > > > > > # cluster node properties (only configure for cluster nodes) # > > > > nifi.cluster.is.node=true > > > > nifi.cluster.node.address=nifi-node1.my.vip.for.dcos > > > > nifi.cluster.node.protocol.port=9001 > > > > nifi.cluster.node.protocol.threads=10 > > > > nifi.cluster.node.protocol.max.threads=50 > > > > nifi.cluster.node.event.history.size=25 > > > > nifi.cluster.node.connection.timeout=30 sec > > > > nifi.cluster.node.read.timeout=30 sec > > > > nifi.cluster.node.max.concurrent.requests=100 > > > > nifi.cluster.firewall.file= > > > > nifi.cluster.flow.election.max.wait.time=1 mins > > > > nifi.cluster.flow.election.max.candidates= > > > > > > # zookeeper properties, used for cluster management # > > > > nifi.zookeeper.connect.string=zookeeper.my.vip.for.dcos:2181 > > > > nifi.zookeeper.connect.timeout=3 secs > > > > nifi.zookeeper.session.timeout=3 secs > > > > nifi.zookeeper.root.node=/some-unique-id > > > > > > > > My suspicion why this is happening is due to the “nifi.web.http.host=” > > configuration does not have a discoverable address and I am not sure > what I > > can configure this to, to make this work. I have configured a VIP for > port > > 80 so that it can be externalized (and load balanced) and be reached > outside > > of the DCOS cluster via web browser (which works for accessing the UI). > It > > seems that the UI replication relies on the web address of the other > nodes > > (I was thinking that this was all done via the nifi.cluster.node.address > and > > nifi.cluster.node.protocol.port properties, but I may have been wrong. > What > > else is the value of “nifi.web.http.host=” used for? > > > > > > Any ideas or any clarity that can be shed on this would be greatly > > appreciated! > > > > > > > > Thanks in Advance, > > > > > > Ryan H. > > > > >
