For Site-to-Site to work between NiFi instances, the following need to be in place:
1. Firewalls have to be open to allow all source nodes to talk to all destination nodes through the Site-To-Site port configured in the target nodes nifi.properties file. 2. Firewalls have to be open to allow all source NiFi Nodes and NCM to talk to HTTP(s) port of the target NCM. 3. In most cases it is necessary to have the nifi.remote.input.socket.host= configured on every target node/NCM with a hostname that is resolveable by the source NiFi nodes/NCM. If left blank the target nodes will use java to try to determine the hostname. Sometimes this can result in localhost being sent which will not work. iT may also return a local hostname which the source NiFi Nodes can not resolve. When setting up your Nifi instances (Nodes or NCMs), it is highly recommended that the following optional properties are populated: # Site to Site properties nifi.remote.input.socket.host= web properties # nifi.web.http.host= (if running non secure) nifi.web.https.host= (if running secure) Thanks, Matt On Thu, Jan 21, 2016 at 12:06 PM, Paresh Shah <[email protected]> wrote: > When the RemoteProcessorGroup has been EnabledForTransmission, the > exception seen in the log file is > > > 2016-01-21 09:01:05,477 WARN [NiFi Site-to-Site Connection Pool > Maintenance] o.a.n.r.c.socket.EndpointConnectionPool > EndpointConnectionPool[Cluster URL=http://localhost:9080/nifi] Unable to > refresh Remote Group's peers due to {} > > java.net.SocketTimeoutException: Timed out reading from socket > > at > org.apache.nifi.remote.io.socket.SocketChannelInputStream.read(SocketChannelInputStream.java:117) > ~[nifi-utils-0.3.0.jar:0.3.0] > > at > org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:51) > ~[nifi-utils-0.3.0.jar:0.3.0] > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > ~[na:1.7.0_79] > > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) > ~[na:1.7.0_79] > > at > org.apache.nifi.remote.io.InterruptableInputStream.read(InterruptableInputStream.java:39) > ~[nifi-utils-0.3.0.jar:0.3.0] > > at java.io.DataInputStream.readInt(DataInputStream.java:387) ~[na:1.7.0_79] > > at > org.apache.nifi.remote.protocol.socket.SocketClientProtocol.getPeerStatuses(SocketClientProtocol.java:237) > ~[nifi-site-to-site-client-0.3.0.jar:0.3.0] > > at > org.apache.nifi.remote.client.socket.EndpointConnectionPool.fetchRemotePeerStatuses(EndpointConnectionPool.java:607) > [nifi-site-to-site-client-0.3.0.jar:0.3.0] > > at > org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshPeers(EndpointConnectionPool.java:839) > [nifi-site-to-site-client-0.3.0.jar:0.3.0] > > at > org.apache.nifi.remote.client.socket.EndpointConnectionPool.access$000(EndpointConnectionPool.java:93) > [nifi-site-to-site-client-0.3.0.jar:0.3.0] > > at > org.apache.nifi.remote.client.socket.EndpointConnectionPool$2.run(EndpointConnectionPool.java:192) > [nifi-site-to-site-client-0.3.0.jar:0.3.0] > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_79] > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > [na:1.7.0_79] > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > [na:1.7.0_79] > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [na:1.7.0_79] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_79] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_79] > > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] > > > --- > > > After about 5 min I see that it was able to send the data over to the > InputPort. I see the yield duration is only 10s and also the input Port is > already started. > > > Paresh > > > > From: Paresh Shah <[email protected]<mailto: > [email protected]>> > Date: Wednesday, January 20, 2016 at 6:12 PM > To: "[email protected]<mailto:[email protected]>" <[email protected] > <mailto:[email protected]>> > Subject: Re: How to configure/start multiple Input ports. > > Another question. > > To use the Remote Processor Group, on the sender side should we do the > following. > > 1. Create Output Port. > 2. Create Remote Processor Group using the OutputPort created. > > Or > > 1. Create the target pipeline with the input port > 2. Create Remote Processor Group with the input port -> This gives > the error that it has timed out reading from the port. But we need to be > sending the data to this port rather then read from it. > > Thanks > Paresh > > From: Paresh Shah <[email protected]<mailto: > [email protected]>> > Date: Wednesday, January 20, 2016 at 4:01 PM > To: "[email protected]<mailto:[email protected]>" <[email protected] > <mailto:[email protected]>> > Subject: Re: How to configure/start multiple Input ports. > > Still don’t see the images. > > Also if the Input Ports are at the root canvas, we are not able to connect > it to the ProcessorGroup. We have all our pipelines within different > processor groups. > > Paresh > > From: Matthew Clarke <[email protected]<mailto: > [email protected]>> > Reply-To: "[email protected]<mailto:[email protected]>" < > [email protected]<mailto:[email protected]>> > Date: Wednesday, January 20, 2016 at 2:37 PM > To: "[email protected]<mailto:[email protected]>" <[email protected] > <mailto:[email protected]>> > Subject: Re: How to configure/start multiple Input ports. > > Three images attached: > > On Wed, Jan 20, 2016 at 4:31 PM, Paresh Shah <[email protected] > <mailto:[email protected]>> wrote: > I cannot see the pictures. Can you please resend it with attachements. > > Thanks > Paresh > > From: Matthew Clarke <[email protected]<mailto: > [email protected]><mailto:[email protected]<mailto: > [email protected]>>> > Reply-To: "[email protected]<mailto:[email protected]><mailto: > [email protected]<mailto:[email protected]>>" <[email protected] > <mailto:[email protected]><mailto:[email protected]<mailto: > [email protected]>>> > Date: Wednesday, January 20, 2016 at 12:55 PM > To: "[email protected]<mailto:[email protected]><mailto: > [email protected]<mailto:[email protected]>>" <[email protected] > <mailto:[email protected]><mailto:[email protected]<mailto: > [email protected]>>> > Subject: Re: How to configure/start multiple Input ports. > > Paresh, > All Site-to-Site data comes across a single port connection. When > adding a RPG to the source NiFi, the URL you provide will be the URL for > the NCM of your internal NiFi cluster. You will add a a different RPG for > each cluster you are connecting to. so in your case the external facing > NiFi cluster will have 2 RPGs on the graph with 1 configured form the URL > of the NCM on internal NiFi cluster 1 and the other for the NCM of internal > NiFi cluster 2. Once a connection is established to an NCM, the NCM will > communicate to the RPG the URLs and site-to-site ports for each of the > clusters connected Nodes as well as the current load on those nodes. The > NiFi with the RPG will then do a smart load-balanced delivery of data to > those nodes. On the target cluster you will add input ports at the root > level of the graph (They cannot be nested inside of any process groups) . > You can add as many uniquely named input ports as you would like. These > input ports will be exposed to the RPG on the sending system. When you > draw a connection to the RPG, you will be provided witha pull-down > selection menu of all available ports. Selecting one will complete this > connection. So all data between NiFi systems via Site-to-Site will use the > same site-to-site port but as you can see it enters the flow on the > destination system via input ports. > > [Inline image 1] > > So on your externally visible NiFi you would have two RPGs setup like > below: > [Inline image 3] > > The receiving NiFi cluster will have one or more uniquely named input > ports: > > [Inline image 4] > > As you can see from the above two screenshots the sending system's > connection to the RPG shows that the connection as being connected to the > input port "Node-Input" on the receiving cluster. > > > Thanks, > Matt > > > > On Wed, Jan 20, 2016 at 2:59 PM, Paresh Shah <[email protected] > <mailto:[email protected]><mailto:[email protected]<mailto: > [email protected]>>> wrote: > > We are trying to see how to run our pipelines in a clustered env using > site-to-site. Our scenario is something like the following. > > 1. 3 different clusters. > 2. One of the clusters is externally visible and will be the primary > cluster. > 3. All the data receivers will run on the primary node. > > Each of the pipelines has a RemoteProcessorGroup( RPG ) that would in > –turn forward the data to their corresponding Input port. > > Looking at the way the site-to-site configuration indicates that we are > only able to specify one port that would be started on a particular node. > So the question is how can we pin a particular RPG to it corresponding > InputPort. > > Any insights would be greatly appreciated. > > Thanks > Paresh > ________________________________ > The information contained in this transmission may contain privileged and > confidential information. It is intended only for the use of the person(s) > named above. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution or duplication of this > communication is strictly prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > ________________________________ > > ________________________________ > The information contained in this transmission may contain privileged and > confidential information. It is intended only for the use of the person(s) > named above. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution or duplication of this > communication is strictly prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > ________________________________ > > ________________________________ > The information contained in this transmission may contain privileged and > confidential information. It is intended only for the use of the person(s) > named above. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution or duplication of this > communication is strictly prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > ________________________________ >
