Hi Dan, Thanks for the info on the changes you made to the Docker image. I am essentially doing the same thing as I have built our own Docker and have a wrapper script that provides the ability to configure the nifi.properties file based on env variables at run time. The problem I am facing is that whatever is configured as the web.host is what will be advertised to the other nodes in the cluster (from what I understand) as well as what Jetty will use to bind to. In our case we need to have two addresses: one for Jetty to bind to (the container localhost) and another for an advertised address for other nodes in the cluster to use (in our case a configured VIP) for the UI/rest api calls. Hope this makes sense.
Cheers, Ryan H On Tue, Jan 30, 2018 at 11:54 AM, Daniel Chaffelson <chaffel...@gmail.com> wrote: > Hi Ryan, > I have proposed a small change to the Docker image which may help you here > - https://github.com/apache/nifi/pull/2439 > Essentially it exposes the port and hostname to be used within > nifi.properties as environment variables which you can pass in at runtime. > Perhaps the approach used will assist you. > > Thanks, > Dan > > On Mon, Jan 29, 2018 at 8:04 PM Ryan H <ryan.howell.developm...@gmail.com> > wrote: > >> Good deal, thanks for getting back to me about it. I think that I may >> make some changes and potentially open a PR depending on what comes of it >> (I'll have to take a look at the workflow and all of that good stuff first). >> >> >> Cheers, >> >> Ryan H >> >> On Mon, Jan 29, 2018 at 12:55 PM, Bryan Bende <bbe...@gmail.com> wrote: >> >>> Ryan, >>> >>> I'm not that familiar with Docker and DCOS, but I think what you said >>> is correct... >>> >>> The issue is that you currently can't leave nifi.web.http.host blank >>> because that will cause the "node API address" of each node to be >>> calculated as 'localhost', which then means replication of requests >>> fails. >>> >>> So you have to set nifi.web.http.host to something that each node can >>> reach. I'm not familiar enough with your setup to know if there is a >>> way to do that. >>> >>> -Bryan >>> >>> >>> On Mon, Jan 29, 2018 at 12:39 PM, Ryan H >>> <ryan.howell.developm...@gmail.com> wrote: >>> > Hi Bryan, >>> > >>> > Yes that makes total sense, and it is what I figured was happening. So >>> > whatever is configured for nifi.web.http.host is where api calls will >>> go to, >>> > but this is also what jetty will bind to, correct? So in my case, I >>> would >>> > have to have the additional property mentioned in >>> > https://issues.apache.org/jira/browse/NIFI-3642 since jetty can't >>> bind to a >>> > VIP. >>> > >>> > -Ryan H >>> > >>> > On Mon, Jan 29, 2018 at 12:28 PM, Bryan Bende <bbe...@gmail.com> >>> wrote: >>> >> >>> >> Ryan, >>> >> >>> >> I remember creating an issue for something that seems similar to what >>> >> you are running into: >>> >> >>> >> https://issues.apache.org/jira/browse/NIFI-3642 >>> >> >>> >> Long story short, I believe you do need to specify a value for >>> >> nifi.web.http.host because that will be used to replicate requests >>> >> that come in to the REST API, so each node needs that value to be >>> >> something that is reachable by the other nodes. >>> >> >>> >> -Bryan >>> >> >>> >> >>> >> On Mon, Jan 29, 2018 at 12:03 PM, Ryan H >>> >> <ryan.howell.developm...@gmail.com> wrote: >>> >> > Dev Team, >>> >> > >>> >> > >>> >> > I am running into an interesting issue while trying to cluster NiFi >>> in a >>> >> > containerized environment (Docker containers running on DC/OS >>> cluster) >>> >> > and I >>> >> > am somewhat stuck with what to do. I am starting with getting just 2 >>> >> > NiFi >>> >> > nodes with a single external zookeeper instance (just to get it >>> working, >>> >> > will not use for production). Currently our DC/OS cluster does not >>> >> > support >>> >> > container-to-container communication (no overlay network support at >>> the >>> >> > moment) so we are using VIP’s to expose required ports on the >>> >> > container(s) >>> >> > so traffic can be mapped to a well known address and correct >>> container >>> >> > port >>> >> > even though the host/host port may change. >>> >> > >>> >> > >>> >> > Currently everything spins up and the UI can be accessed on whatever >>> >> > node is >>> >> > elected the Cluster Coordinator (in this case it is the Primary >>> Node as >>> >> > well) (does show that there are 2 nodes in the cluster). However, >>> any >>> >> > action >>> >> > taken on the canvas results in the following error shown in the UI: >>> >> > >>> >> > >>> >> > Node localhost:80 is unable to fulfill this request due to: >>> Transaction >>> >> > c91764e4-2fc8-492b-8887-babb59981ff3 is already in progress. >>> >> > >>> >> > >>> >> > When trying to access the UI of the other node, the canvas cannot be >>> >> > reached >>> >> > and the following error is shown on the error splash screen >>> (increased >>> >> > read >>> >> > timeout to 30 secs, still the same): >>> >> > >>> >> > >>> >> > An unexpected error has occurred >>> >> > com.sun.jersey.api.client.ClientHandlerException: >>> >> > java.net.SocketTimeoutException: Read timed out >>> >> > >>> >> > >>> >> > If configured to use the hostname of the container, then the error >>> is: >>> >> > >>> >> > unknown host exception >>> >> > >>> >> > >>> >> > In the NiFi logs, the following errors are present (as well as some >>> >> > other >>> >> > warnings): >>> >> > >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] >>> >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate >>> request >>> >> > GET >>> >> > /nifi-api/flow/current-user to localhost:80 due to >>> >> > com.sun.jersey.api.client.ClientHandlerException: >>> >> > java.net.SocketTimeoutException: Read timed out >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 14:46:52,393 WARN [Replicate Request Thread-3] >>> >> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > com.sun.jersey.api.client.ClientHandlerException: >>> >> > java.net.SocketTimeoutException: Read timed out >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler. >>> handle(URLConnectionClientHandler.java:155) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at com.sun.jersey.api.client.Client.handle(Client.java:652) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle( >>> GZIPContentEncodingFilter.java:123) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at com.sun.jersey.api.client.WebResource.handle( >>> WebResource.java:682) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at com.sun.jersey.api.client.WebResource.access$200( >>> WebResource.java:74) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > com.sun.jersey.api.client.WebResource$Builder.get( >>> WebResource.java:509) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > org.apache.nifi.cluster.coordination.http.replication. >>> ThreadPoolRequestReplicator.replicateRequest( >>> ThreadPoolRequestReplicator.java:641) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > org.apache.nifi.cluster.coordination.http.replication. >>> ThreadPoolRequestReplicator$NodeHttpRequest.run( >>> ThreadPoolRequestReplicator.java:852) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > java.util.concurrent.Executors$RunnableAdapter. >>> call(Executors.java:511) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > java.util.concurrent.ThreadPoolExecutor.runWorker( >>> ThreadPoolExecutor.java:1149) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run( >>> ThreadPoolExecutor.java:624) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.lang.Thread.run(Thread.java:748) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > Caused by: java.net.SocketTimeoutException: Read timed out >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.net.SocketInputStream.socketRead0(Native Method) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.net.SocketInputStream.socketRead(SocketInputStream. >>> java:116) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.net.SocketInputStream.read(SocketInputStream.java:171) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.net.SocketInputStream.read(SocketInputStream.java:141) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0( >>> HttpURLConnection.java:1587) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream( >>> HttpURLConnection.java:1492) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > java.net.HttpURLConnection.getResponseCode( >>> HttpURLConnection.java:480) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler._ >>> invoke(URLConnectionClientHandler.java:253) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler. >>> handle(URLConnectionClientHandler.java:153) >>> >> > >>> >> > 2018-01-29 14:46:52,393 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > ... 12 common frames omitted >>> >> > >>> >> > >>> >> > >>> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:42,827 WARN [Process Cluster Protocol Request-4] >>> >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol >>> message >>> >> > from <ip-host.aws.internal.address> due to java.io.EOFException >>> >> > >>> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > java.io.EOFException: null >>> >> > >>> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2. >>> unmarshal(JaxbProtocolContext.java:112) >>> >> > >>> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > org.apache.nifi.cluster.protocol.impl.SocketProtocolListener. >>> dispatchRequest(SocketProtocolListener.java:142) >>> >> > >>> >> > 2018-01-29 15:19:42,827 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > org.apache.nifi.io.socket.SocketListener$2$1.run( >>> SocketListener.java:136) >>> >> > >>> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > java.util.concurrent.ThreadPoolExecutor.runWorker( >>> ThreadPoolExecutor.java:1149) >>> >> > >>> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at >>> >> > >>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run( >>> ThreadPoolExecutor.java:624) >>> >> > >>> >> > 2018-01-29 15:19:42,828 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > at java.lang.Thread.run(Thread.java:748) >>> >> > >>> >> > 2018-01-29 15:19:45,505 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:45,505 INFO [Process Cluster Protocol Request-5] >>> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >>> >> > 181a3bd2-894e-48bf-a8d3-c298d1992524 (type=HEARTBEAT, length=2555 >>> bytes) >>> >> > from localhost:80 in 2 millis >>> >> > >>> >> > 2018-01-29 15:19:46,118 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:46,118 INFO [Process Cluster Protocol Request-6] >>> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >>> >> > f864bf7b-f8fc-40e9-8f04-0cd9d96b3f66 (type=HEARTBEAT, length=2555 >>> bytes) >>> >> > from localhost:80 in 2 millis >>> >> > >>> >> > 2018-01-29 15:19:46,119 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:46,119 INFO [Clustering Tasks Thread-1] >>> >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at >>> 2018-01-29 >>> >> > 15:19:46,114 and sent to nifi-node1.my.vip.for.dcos:9001 at >>> 2018-01-29 >>> >> > 15:19:46,119; send took 4 millis >>> >> > >>> >> > 2018-01-29 15:19:47,599 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:47,599 INFO [Heartbeat Monitor Thread-1] >>> >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 >>> heartbeats in >>> >> > 7490 nanos >>> >> > >>> >> > 2018-01-29 15:19:50,515 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:50,515 INFO [Process Cluster Protocol Request-7] >>> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >>> >> > d9577954-71e3-41eb-b956-c235a98bf6f9 (type=HEARTBEAT, length=2555 >>> bytes) >>> >> > from localhost:80 in 4 millis >>> >> > >>> >> > 2018-01-29 15:19:51,122 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:51,122 INFO [Process Cluster Protocol Request-8] >>> >> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request >>> >> > 18626790-5d19-44ab-af0e-b3246d0a4482 (type=HEARTBEAT, length=2555 >>> bytes) >>> >> > from localhost:80 in 2 millis >>> >> > >>> >> > 2018-01-29 15:19:51,123 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:51,123 INFO [Clustering Tasks Thread-1] >>> >> > o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at >>> 2018-01-29 >>> >> > 15:19:51,119 and sent to nifi-node1.my.vip.for.dcos:9001 at >>> 2018-01-29 >>> >> > 15:19:51,123; send took 4 millis >>> >> > >>> >> > 2018-01-29 15:19:52,601 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:52,600 INFO [Heartbeat Monitor Thread-1] >>> >> > o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 >>> heartbeats in >>> >> > 7776 nanos >>> >> > >>> >> > 2018-01-29 15:19:52,828 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > 2018-01-29 15:19:52,828 WARN [Process Cluster Protocol Request-9] >>> >> > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol >>> message >>> >> > from <ip-host.aws.internal.address> due to java.io.EOFException >>> >> > >>> >> > 2018-01-29 15:19:52,829 INFO [NiFi logging handler] >>> >> > org.apache.nifi.StdOut >>> >> > java.io.EOFException: null >>> >> > >>> >> > >>> >> > >>> >> > Here are relevant configs in nifi.properties file: >>> >> > >>> >> > # web properties # >>> >> > >>> >> > nifi.web.war.directory=./lib >>> >> > >>> >> > # for web host, I have tried: localhost, container hostname, left >>> blank >>> >> > (to >>> >> > listen on all interfaces), the VIP configured for external access >>> all of >>> >> > which have failed (read timeout or unknown host exception) >>> >> > >>> >> > nifi.web.http.host= >>> >> > >>> >> > nifi.web.http.port=80 >>> >> > >>> >> > nifi.web.http.network.interface.default= >>> >> > >>> >> > nifi.web.https.host= >>> >> > >>> >> > nifi.web.https.port= >>> >> > >>> >> > nifi.web.https.network.interface.default= >>> >> > >>> >> > nifi.web.jetty.working.directory=./work/jetty >>> >> > >>> >> > nifi.web.jetty.threads=200 >>> >> > >>> >> > >>> >> > # cluster node properties (only configure for cluster nodes) # >>> >> > >>> >> > nifi.cluster.is.node=true >>> >> > >>> >> > nifi.cluster.node.address=nifi-node1.my.vip.for.dcos >>> >> > >>> >> > nifi.cluster.node.protocol.port=9001 >>> >> > >>> >> > nifi.cluster.node.protocol.threads=10 >>> >> > >>> >> > nifi.cluster.node.protocol.max.threads=50 >>> >> > >>> >> > nifi.cluster.node.event.history.size=25 >>> >> > >>> >> > nifi.cluster.node.connection.timeout=30 sec >>> >> > >>> >> > nifi.cluster.node.read.timeout=30 sec >>> >> > >>> >> > nifi.cluster.node.max.concurrent.requests=100 >>> >> > >>> >> > nifi.cluster.firewall.file= >>> >> > >>> >> > nifi.cluster.flow.election.max.wait.time=1 mins >>> >> > >>> >> > nifi.cluster.flow.election.max.candidates= >>> >> > >>> >> > >>> >> > # zookeeper properties, used for cluster management # >>> >> > >>> >> > nifi.zookeeper.connect.string=zookeeper.my.vip.for.dcos:2181 >>> >> > >>> >> > nifi.zookeeper.connect.timeout=3 secs >>> >> > >>> >> > nifi.zookeeper.session.timeout=3 secs >>> >> > >>> >> > nifi.zookeeper.root.node=/some-unique-id >>> >> > >>> >> > >>> >> > >>> >> > My suspicion why this is happening is due to the >>> “nifi.web.http.host=” >>> >> > configuration does not have a discoverable address and I am not sure >>> >> > what I >>> >> > can configure this to, to make this work. I have configured a VIP >>> for >>> >> > port >>> >> > 80 so that it can be externalized (and load balanced) and be reached >>> >> > outside >>> >> > of the DCOS cluster via web browser (which works for accessing the >>> UI). >>> >> > It >>> >> > seems that the UI replication relies on the web address of the other >>> >> > nodes >>> >> > (I was thinking that this was all done via the >>> nifi.cluster.node.address >>> >> > and >>> >> > nifi.cluster.node.protocol.port properties, but I may have been >>> wrong. >>> >> > What >>> >> > else is the value of “nifi.web.http.host=” used for? >>> >> > >>> >> > >>> >> > Any ideas or any clarity that can be shed on this would be greatly >>> >> > appreciated! >>> >> > >>> >> > >>> >> > >>> >> > Thanks in Advance, >>> >> > >>> >> > >>> >> > Ryan H. >>> >> > >>> >> > >>> > >>> > >>> >> >>
