Hi Ryan, I have proposed a small change to the Docker image which may help you here - https://github.com/apache/nifi/pull/2439 Essentially it exposes the port and hostname to be used within nifi.properties as environment variables which you can pass in at runtime. Perhaps the approach used will assist you.
Thanks, Dan On Mon, Jan 29, 2018 at 8:04 PM Ryan H <[email protected]> wrote: > Good deal, thanks for getting back to me about it. I think that I may make > some changes and potentially open a PR depending on what comes of it (I'll > have to take a look at the workflow and all of that good stuff first). > > > Cheers, > > Ryan H > > On Mon, Jan 29, 2018 at 12:55 PM, Bryan Bende <[email protected]> wrote: > >> Ryan, >> >> I'm not that familiar with Docker and DCOS, but I think what you said >> is correct... >> >> The issue is that you currently can't leave nifi.web.http.host blank >> because that will cause the "node API address" of each node to be >> calculated as 'localhost', which then means replication of requests >> fails. >> >> So you have to set nifi.web.http.host to something that each node can >> reach. I'm not familiar enough with your setup to know if there is a >> way to do that. >> >> -Bryan >> >> >> On Mon, Jan 29, 2018 at 12:39 PM, Ryan H >> <[email protected]> wrote: >> > Hi Bryan, >> > >> > Yes that makes total sense, and it is what I figured was happening. So >> > whatever is configured for nifi.web.http.host is where api calls will >> go to, >> > but this is also what jetty will bind to, correct? So in my case, I >> would >> > have to have the additional property mentioned in >> > https://issues.apache.org/jira/browse/NIFI-3642 since jetty can't bind >> to a >> > VIP. >> > >> > -Ryan H >> > >> > On Mon, Jan 29, 2018 at 12:28 PM, Bryan Bende <[email protected]> wrote: >> >> >> >> Ryan, >> >> >> >> I remember creating an issue for something that seems similar to what >> >> you are running into: >> >> >> >> https://issues.apache.org/jira/browse/NIFI-3642 >> >> >> >> Long story short, I believe you do need to specify a value for >> >> nifi.web.http.host because that will be used to replicate requests >> >> that come in to the REST API, so each node needs that value to be >> >> something that is reachable by the other nodes. >> >> >> >> -Bryan >> >> >> >> >> >> On Mon, Jan 29, 2018 at 12:03 PM, Ryan H >> >> <[email protected]> wrote: >> >> > Dev Team, >> >> > >> >> > >> >> > I am running into an interesting issue while trying to cluster NiFi >> in a >> >> > containerized environment (Docker containers running on DC/OS >> cluster) >> >> > and I >> >> > am somewhat stuck with what to do. I am starting with getting just 2 >> >> > NiFi >> >> > nodes with a single external zookeeper instance (just to get it >> working, >> >> > will not use for production). Currently our DC/OS cluster does not >> >> > support >> >> > container-to-container communication (no overlay network support at >> the >> >> > moment) so we are using VIP’s to expose required ports on the >> >> > container(s) >> >> > so traffic can be mapped to a well known address and correct >> container >> >> > port >> >> > even though the host/host port may change. >> >> > >> >> > >> >> > Currently everything spins up and the UI can be accessed on whatever >> >> > node is >> >> > elected the Cluster Coordinator (in this case it is the Primary Node >> as >> >> > well) (does show that there are 2 nodes in the cluster). However, any >> >> > action >> >> > taken on the canvas results in the following error shown in the UI: >> >> > >> >> > >> >> > Node localhost:80 is unable to fulfill this request due to: >> Transaction >> >> > c91764e4-2fc8-492b-8887-babb59981ff3 is already in progress. >> >> > >> >> > >> >> > When trying to access the UI of the other node, the canvas cannot be >> >> > reached >> >> > and the following error is shown on the error splash screen >> (increased >> >> > read >> >> > timeout to 30 secs, still the same): >> >> > >> >> > >> >> > An unexpected error has occurred >> >> > com.sun.jersey.api.client.ClientHandlerException: >> >> > java.net.SocketTimeoutException: Read timed out >> >> > >> >> > >> >> > If configured to use the hostname of the container, then the error >> is: >> >> > >> >> > unknown host exception >> >> > >> >> > >> >> > In the NiFi logs, the following errors are present (as well as some >> >> > other >> >> > warnings): >> >> > >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] >> >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request >> >> > GET >> >> > /nifi-api/flow/current-user to localhost:80 due to >> >> > com.sun.jersey.api.client.ClientHandlerException: >> >> > java.net.SocketTimeoutException: Read timed out >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] >> >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > com.sun.jersey.api.client.ClientHandlerException: >> >> > java.net.SocketTimeoutException: Read timed out >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at com.sun.jersey.api.client.Client.handle(Client.java:652) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:641) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:852) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.lang.Thread.run(Thread.java:748) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > Caused by: java.net.SocketTimeoutException: Read timed out >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.net.SocketInputStream.socketRead0(Native Method) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.net.SocketInputStream.read(SocketInputStream.java:171) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.net.SocketInputStream.read(SocketInputStream.java:141) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) >> >> > >> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > ... 12 common frames omitted >> >> > >> >> > >> >> > >> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:42,827 WARN [Process Cluster Protocol Request-4] >> >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol >> message >> >> > from <ip-host.aws.internal.address> due to java.io.EOFException >> >> > >> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > java.io.EOFException: null >> >> > >> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:112) >> >> > >> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:142) >> >> > >> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) >> >> > >> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> > >> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at >> >> > >> >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> > >> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > at java.lang.Thread.run(Thread.java:748) >> >> > >> >> > 2018-01-29 15:19:45,505 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:45,505 INFO [Process Cluster Protocol Request-5] >> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >> >> > 181a3bd2-894e-48bf-a8d3-c298d1992524 (type=HEARTBEAT, length=2555 >> bytes) >> >> > from localhost:80 in 2 millis >> >> > >> >> > 2018-01-29 15:19:46,118 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:46,118 INFO [Process Cluster Protocol Request-6] >> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >> >> > f864bf7b-f8fc-40e9-8f04-0cd9d96b3f66 (type=HEARTBEAT, length=2555 >> bytes) >> >> > from localhost:80 in 2 millis >> >> > >> >> > 2018-01-29 15:19:46,119 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:46,119 INFO [Clustering Tasks Thread-1] >> >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 >> >> > 15:19:46,114 and sent to nifi-node1.my.vip.for.dcos:9001 at >> 2018-01-29 >> >> > 15:19:46,119; send took 4 millis >> >> > >> >> > 2018-01-29 15:19:47,599 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:47,599 INFO [Heartbeat Monitor Thread-1] >> >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 >> heartbeats in >> >> > 7490 nanos >> >> > >> >> > 2018-01-29 15:19:50,515 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:50,515 INFO [Process Cluster Protocol Request-7] >> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >> >> > d9577954-71e3-41eb-b956-c235a98bf6f9 (type=HEARTBEAT, length=2555 >> bytes) >> >> > from localhost:80 in 4 millis >> >> > >> >> > 2018-01-29 15:19:51,122 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:51,122 INFO [Process Cluster Protocol Request-8] >> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >> >> > 18626790-5d19-44ab-af0e-b3246d0a4482 (type=HEARTBEAT, length=2555 >> bytes) >> >> > from localhost:80 in 2 millis >> >> > >> >> > 2018-01-29 15:19:51,123 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:51,123 INFO [Clustering Tasks Thread-1] >> >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-29 >> >> > 15:19:51,119 and sent to nifi-node1.my.vip.for.dcos:9001 at >> 2018-01-29 >> >> > 15:19:51,123; send took 4 millis >> >> > >> >> > 2018-01-29 15:19:52,601 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:52,600 INFO [Heartbeat Monitor Thread-1] >> >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 >> heartbeats in >> >> > 7776 nanos >> >> > >> >> > 2018-01-29 15:19:52,828 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > 2018-01-29 15:19:52,828 WARN [Process Cluster Protocol Request-9] >> >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol >> message >> >> > from <ip-host.aws.internal.address> due to java.io.EOFException >> >> > >> >> > 2018-01-29 15:19:52,829 INFO [NiFi logging handler] >> >> > org.apache.nifi.StdOut >> >> > java.io.EOFException: null >> >> > >> >> > >> >> > >> >> > Here are relevant configs in nifi.properties file: >> >> > >> >> > # web properties # >> >> > >> >> > nifi.web.war.directory=./lib >> >> > >> >> > # for web host, I have tried: localhost, container hostname, left >> blank >> >> > (to >> >> > listen on all interfaces), the VIP configured for external access >> all of >> >> > which have failed (read timeout or unknown host exception) >> >> > >> >> > nifi.web.http.host= >> >> > >> >> > nifi.web.http.port=80 >> >> > >> >> > nifi.web.http.network.interface.default= >> >> > >> >> > nifi.web.https.host= >> >> > >> >> > nifi.web.https.port= >> >> > >> >> > nifi.web.https.network.interface.default= >> >> > >> >> > nifi.web.jetty.working.directory=./work/jetty >> >> > >> >> > nifi.web.jetty.threads=200 >> >> > >> >> > >> >> > # cluster node properties (only configure for cluster nodes) # >> >> > >> >> > nifi.cluster.is.node=true >> >> > >> >> > nifi.cluster.node.address=nifi-node1.my.vip.for.dcos >> >> > >> >> > nifi.cluster.node.protocol.port=9001 >> >> > >> >> > nifi.cluster.node.protocol.threads=10 >> >> > >> >> > nifi.cluster.node.protocol.max.threads=50 >> >> > >> >> > nifi.cluster.node.event.history.size=25 >> >> > >> >> > nifi.cluster.node.connection.timeout=30 sec >> >> > >> >> > nifi.cluster.node.read.timeout=30 sec >> >> > >> >> > nifi.cluster.node.max.concurrent.requests=100 >> >> > >> >> > nifi.cluster.firewall.file= >> >> > >> >> > nifi.cluster.flow.election.max.wait.time=1 mins >> >> > >> >> > nifi.cluster.flow.election.max.candidates= >> >> > >> >> > >> >> > # zookeeper properties, used for cluster management # >> >> > >> >> > nifi.zookeeper.connect.string=zookeeper.my.vip.for.dcos:2181 >> >> > >> >> > nifi.zookeeper.connect.timeout=3 secs >> >> > >> >> > nifi.zookeeper.session.timeout=3 secs >> >> > >> >> > nifi.zookeeper.root.node=/some-unique-id >> >> > >> >> > >> >> > >> >> > My suspicion why this is happening is due to the >> “nifi.web.http.host=” >> >> > configuration does not have a discoverable address and I am not sure >> >> > what I >> >> > can configure this to, to make this work. I have configured a VIP for >> >> > port >> >> > 80 so that it can be externalized (and load balanced) and be reached >> >> > outside >> >> > of the DCOS cluster via web browser (which works for accessing the >> UI). >> >> > It >> >> > seems that the UI replication relies on the web address of the other >> >> > nodes >> >> > (I was thinking that this was all done via the >> nifi.cluster.node.address >> >> > and >> >> > nifi.cluster.node.protocol.port properties, but I may have been >> wrong. >> >> > What >> >> > else is the value of “nifi.web.http.host=” used for? >> >> > >> >> > >> >> > Any ideas or any clarity that can be shed on this would be greatly >> >> > appreciated! >> >> > >> >> > >> >> > >> >> > Thanks in Advance, >> >> > >> >> > >> >> > Ryan H. >> >> > >> >> > >> > >> > >> > >
