Good deal, thanks for getting back to me about it. I think that I may make some changes and potentially open a PR depending on what comes of it (I'll have to take a look at the workflow and all of that good stuff first).
Cheers, Ryan H On Mon, Jan 29, 2018 at 12:55 PM, Bryan Bende <bbe...@gmail.com> wrote: > Ryan, > > I'm not that familiar with Docker and DCOS, but I think what you said > is correct... > > The issue is that you currently can't leave nifi.web.http.host blank > because that will cause the "node API address" of each node to be > calculated as 'localhost', which then means replication of requests > fails. > > So you have to set nifi.web.http.host to something that each node can > reach. I'm not familiar enough with your setup to know if there is a > way to do that. > > -Bryan > > > On Mon, Jan 29, 2018 at 12:39 PM, Ryan H > <ryan.howell.developm...@gmail.com> wrote: > > Hi Bryan, > > > > Yes that makes total sense, and it is what I figured was happening. So > > whatever is configured for nifi.web.http.host is where api calls will go > to, > > but this is also what jetty will bind to, correct? So in my case, I would > > have to have the additional property mentioned in > > https://issues.apache.org/jira/browse/NIFI-3642 since jetty can't bind > to a > > VIP. > > > > -Ryan H > > > > On Mon, Jan 29, 2018 at 12:28 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> > >> Ryan, > >> > >> I remember creating an issue for something that seems similar to what > >> you are running into: > >> > >> https://issues.apache.org/jira/browse/NIFI-3642 > >> > >> Long story short, I believe you do need to specify a value for > >> nifi.web.http.host because that will be used to replicate requests > >> that come in to the REST API, so each node needs that value to be > >> something that is reachable by the other nodes. > >> > >> -Bryan > >> > >> > >> On Mon, Jan 29, 2018 at 12:03 PM, Ryan H > >> <ryan.howell.developm...@gmail.com> wrote: > >> > Dev Team, > >> > > >> > > >> > I am running into an interesting issue while trying to cluster NiFi > in a > >> > containerized environment (Docker containers running on DC/OS cluster) > >> > and I > >> > am somewhat stuck with what to do. I am starting with getting just 2 > >> > NiFi > >> > nodes with a single external zookeeper instance (just to get it > working, > >> > will not use for production). Currently our DC/OS cluster does not > >> > support > >> > container-to-container communication (no overlay network support at > the > >> > moment) so we are using VIP’s to expose required ports on the > >> > container(s) > >> > so traffic can be mapped to a well known address and correct container > >> > port > >> > even though the host/host port may change. > >> > > >> > > >> > Currently everything spins up and the UI can be accessed on whatever > >> > node is > >> > elected the Cluster Coordinator (in this case it is the Primary Node > as > >> > well) (does show that there are 2 nodes in the cluster). However, any > >> > action > >> > taken on the canvas results in the following error shown in the UI: > >> > > >> > > >> > Node localhost:80 is unable to fulfill this request due to: > Transaction > >> > c91764e4-2fc8-492b-8887-babb59981ff3 is already in progress. > >> > > >> > > >> > When trying to access the UI of the other node, the canvas cannot be > >> > reached > >> > and the following error is shown on the error splash screen (increased > >> > read > >> > timeout to 30 secs, still the same): > >> > > >> > > >> > An unexpected error has occurred > >> > com.sun.jersey.api.client.ClientHandlerException: > >> > java.net.SocketTimeoutException: Read timed out > >> > > >> > > >> > If configured to use the hostname of the container, then the error is: > >> > > >> > unknown host exception > >> > > >> > > >> > In the NiFi logs, the following errors are present (as well as some > >> > other > >> > warnings): > >> > > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] > >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request > >> > GET > >> > /nifi-api/flow/current-user to localhost:80 due to > >> > com.sun.jersey.api.client.ClientHandlerException: > >> > java.net.SocketTimeoutException: Read timed out > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] > >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > com.sun.jersey.api.client.ClientHandlerException: > >> > java.net.SocketTimeoutException: Read timed out > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler. > handle(URLConnectionClientHandler.java:155) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at com.sun.jersey.api.client.Client.handle(Client.java:652) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle( > GZIPContentEncodingFilter.java:123) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at com.sun.jersey.api.client.WebResource.access$200( > WebResource.java:74) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > com.sun.jersey.api.client.WebResource$Builder.get( > WebResource.java:509) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > org.apache.nifi.cluster.coordination.http.replication. > ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator. > java:641) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > org.apache.nifi.cluster.coordination.http.replication. > ThreadPoolRequestReplicator$NodeHttpRequest.run( > ThreadPoolRequestReplicator.java:852) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.lang.Thread.run(Thread.java:748) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > Caused by: java.net.SocketTimeoutException: Read timed out > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.net.SocketInputStream.socketRead0(Native Method) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.net.SocketInputStream.read(SocketInputStream.java:171) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.net.SocketInputStream.read(SocketInputStream.java:141) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0( > HttpURLConnection.java:1587) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream( > HttpURLConnection.java:1492) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > java.net.HttpURLConnection.getResponseCode( > HttpURLConnection.java:480) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler._ > invoke(URLConnectionClientHandler.java:253) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler. > handle(URLConnectionClientHandler.java:153) > >> > > >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > ... 12 common frames omitted > >> > > >> > > >> > > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:42,827 WARN [Process Cluster Protocol Request-4] > >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol > message > >> > from <ip-host.aws.internal.address> due to java.io.EOFException > >> > > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > java.io.EOFException: null > >> > > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2. > unmarshal(JaxbProtocolContext.java:112) > >> > > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > org.apache.nifi.cluster.protocol.impl.SocketProtocolListener. > dispatchRequest(SocketProtocolListener.java:142) > >> > > >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > org.apache.nifi.io.socket.SocketListener$2$1.run( > SocketListener.java:136) > >> > > >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > >> > > >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > >> > > >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > at java.lang.Thread.run(Thread.java:748) > >> > > >> > 2018-01-29 15:19:45,505 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:45,505 INFO [Process Cluster Protocol Request-5] > >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > >> > 181a3bd2-894e-48bf-a8d3-c298d1992524 (type=HEARTBEAT, length=2555 > bytes) > >> > from localhost:80 in 2 millis > >> > > >> > 2018-01-29 15:19:46,118 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:46,118 INFO [Process Cluster Protocol Request-6] > >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > >> > f864bf7b-f8fc-40e9-8f04-0cd9d96b3f66 (type=HEARTBEAT, length=2555 > bytes) > >> > from localhost:80 in 2 millis > >> > > >> > 2018-01-29 15:19:46,119 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:46,119 INFO [Clustering Tasks Thread-1] > >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 > >> > 15:19:46,114 and sent to nifi-node1.my.vip.for.dcos:9001 at > 2018-01-29 > >> > 15:19:46,119; send took 4 millis > >> > > >> > 2018-01-29 15:19:47,599 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:47,599 INFO [Heartbeat Monitor Thread-1] > >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 > heartbeats in > >> > 7490 nanos > >> > > >> > 2018-01-29 15:19:50,515 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:50,515 INFO [Process Cluster Protocol Request-7] > >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > >> > d9577954-71e3-41eb-b956-c235a98bf6f9 (type=HEARTBEAT, length=2555 > bytes) > >> > from localhost:80 in 4 millis > >> > > >> > 2018-01-29 15:19:51,122 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:51,122 INFO [Process Cluster Protocol Request-8] > >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > >> > 18626790-5d19-44ab-af0e-b3246d0a4482 (type=HEARTBEAT, length=2555 > bytes) > >> > from localhost:80 in 2 millis > >> > > >> > 2018-01-29 15:19:51,123 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:51,123 INFO [Clustering Tasks Thread-1] > >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 > >> > 15:19:51,119 and sent to nifi-node1.my.vip.for.dcos:9001 at > 2018-01-29 > >> > 15:19:51,123; send took 4 millis > >> > > >> > 2018-01-29 15:19:52,601 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:52,600 INFO [Heartbeat Monitor Thread-1] > >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 > heartbeats in > >> > 7776 nanos > >> > > >> > 2018-01-29 15:19:52,828 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > 2018-01-29 15:19:52,828 WARN [Process Cluster Protocol Request-9] > >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol > message > >> > from <ip-host.aws.internal.address> due to java.io.EOFException > >> > > >> > 2018-01-29 15:19:52,829 INFO [NiFi logging handler] > >> > org.apache.nifi.StdOut > >> > java.io.EOFException: null > >> > > >> > > >> > > >> > Here are relevant configs in nifi.properties file: > >> > > >> > # web properties # > >> > > >> > nifi.web.war.directory=./lib > >> > > >> > # for web host, I have tried: localhost, container hostname, left > blank > >> > (to > >> > listen on all interfaces), the VIP configured for external access all > of > >> > which have failed (read timeout or unknown host exception) > >> > > >> > nifi.web.http.host= > >> > > >> > nifi.web.http.port=80 > >> > > >> > nifi.web.http.network.interface.default= > >> > > >> > nifi.web.https.host= > >> > > >> > nifi.web.https.port= > >> > > >> > nifi.web.https.network.interface.default= > >> > > >> > nifi.web.jetty.working.directory=./work/jetty > >> > > >> > nifi.web.jetty.threads=200 > >> > > >> > > >> > # cluster node properties (only configure for cluster nodes) # > >> > > >> > nifi.cluster.is.node=true > >> > > >> > nifi.cluster.node.address=nifi-node1.my.vip.for.dcos > >> > > >> > nifi.cluster.node.protocol.port=9001 > >> > > >> > nifi.cluster.node.protocol.threads=10 > >> > > >> > nifi.cluster.node.protocol.max.threads=50 > >> > > >> > nifi.cluster.node.event.history.size=25 > >> > > >> > nifi.cluster.node.connection.timeout=30 sec > >> > > >> > nifi.cluster.node.read.timeout=30 sec > >> > > >> > nifi.cluster.node.max.concurrent.requests=100 > >> > > >> > nifi.cluster.firewall.file= > >> > > >> > nifi.cluster.flow.election.max.wait.time=1 mins > >> > > >> > nifi.cluster.flow.election.max.candidates= > >> > > >> > > >> > # zookeeper properties, used for cluster management # > >> > > >> > nifi.zookeeper.connect.string=zookeeper.my.vip.for.dcos:2181 > >> > > >> > nifi.zookeeper.connect.timeout=3 secs > >> > > >> > nifi.zookeeper.session.timeout=3 secs > >> > > >> > nifi.zookeeper.root.node=/some-unique-id > >> > > >> > > >> > > >> > My suspicion why this is happening is due to the “nifi.web.http.host=” > >> > configuration does not have a discoverable address and I am not sure > >> > what I > >> > can configure this to, to make this work. I have configured a VIP for > >> > port > >> > 80 so that it can be externalized (and load balanced) and be reached > >> > outside > >> > of the DCOS cluster via web browser (which works for accessing the > UI). > >> > It > >> > seems that the UI replication relies on the web address of the other > >> > nodes > >> > (I was thinking that this was all done via the > nifi.cluster.node.address > >> > and > >> > nifi.cluster.node.protocol.port properties, but I may have been > wrong. > >> > What > >> > else is the value of “nifi.web.http.host=” used for? > >> > > >> > > >> > Any ideas or any clarity that can be shed on this would be greatly > >> > appreciated! > >> > > >> > > >> > > >> > Thanks in Advance, > >> > > >> > > >> > Ryan H. > >> > > >> > > > > > >
