Hey Brian, Just wanted to follow up on this and see if you were able to get everything resolved, or if you are still running into problems.
Thanks -Mark ---------------------------------------- > From: [email protected] > To: [email protected] > Subject: RE: NiFi Site-to-Site > Date: Tue, 25 Aug 2015 09:10:53 -0500 > > Brian, > > As part of the site-to-site communication, the client node (i.e., the NiFi > node establishing the connection, > the one with the Remote Process Group) will connect and request a list of > "peers" > (i.e., a list of nodes in the cluster). If you are running a very new version > of 0.3.0-SNAPSHOT, you will > be able to configure what value is returned by that node by setting a value > for the > "nifi.remote.input.socket.host" property in the nifi.properties file. If what > you are running is a bit older, > or if that property is not set, what gets returned is the hostname that is > returned by Java. > > Generally, when you see this error, it is because the hostname returned by > the remote nifi instance > is not in your /etc/hosts file on the client. > > If you run the "hostname" command on the remote instance, can you verify that > the value returned > is in /etc/hosts on the client instance? > > Thanks > -Mark > > ---------------------------------------- >> Date: Mon, 24 Aug 2015 21:41:19 -0400 >> Subject: Re: NiFi Site-to-Site >> From: [email protected] >> To: [email protected] >> >> Hey Mark, >> >> Thanks for the reply and the references. I've taken a look through, and it >> seems like everything should be set up properly. I am seeing the following >> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the >> NiFi Producer instance fine from the Consumer box, but Site-to-Site is >> still giving some trouble. That's why I started looking at AWS Security >> Groups and Firewall settings. Does this stack trace ring a bell with >> anything? >> >> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8] >> o.a.n.c.t.ContinuallyRunConnectableTask >> RemoteGroupPort[name=data-output,target=http://nifi-producer-host:8080/nifi] >> Administratively Pausing for 10 seconds due to processing failure: >> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException >> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException >> at >> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234) >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80) >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40) >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_45] >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> [na:1.8.0_45] >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> [na:1.8.0_45] >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >> [na:1.8.0_45] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_45] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_45] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] >> Caused by: java.nio.channels.UnresolvedAddressException: null >> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45] >> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) >> ~[na:1.8.0_45] >> at java.nio.channels.SocketChannel.open(SocketChannel.java:189) >> ~[na:1.8.0_45] >> at >> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708) >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682) >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300) >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129) >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167) >> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> at >> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227) >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> ... 10 common frames omitted >> >> Thanks again, >> Brian >> >> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <[email protected]> wrote: >> >>> Hey Brian, >>> >>> The latest version of the User Guide has instructions on setting up >>> site-to-site, but that version >>> hasn't yet been posted to the website, I don't believe. If you click the >>> "help" menu in the top-right >>> corner of your NiFi instance, and go to the User Guide from there, you >>> should have the up-to-date >>> guide there. >>> >>> If there's anything missing, please let us know so that we can update the >>> guide. >>> >>> For convenience, I have pasted the text of that section below: >>> >>> >>> In order to communicate with a remote NiFi instance via Site-to-Site, >>> simply drag a Remote Process Group onto the graph and enter the URL of the >>> remote NiFi instance (for more information on the components of a Remote >>> Process Group, see Remote Process Group Transmission section of this >>> guide.) The URL is the same URL you would use to go to that instance’s User >>> Interface. At that point, you can drag a connection to or from the Remote >>> Process Group in the same way you would drag a connection to or from a >>> Processor or a local Process Group. When you drag the connection, you will >>> have a chance to choose which Port to connect to. Note that it may take up >>> to one minute for the Remote Process Group to determine which ports are >>> available. >>> >>> If the connection is dragged starting from the Remote Process Group, the >>> ports shown will be the Output Ports of the remote group, as this indicates >>> that you will be pulling data from the remote instance. If the connection >>> instead ends on the Remote Process Group, the ports shown will be the Input >>> Ports of the remote group, as this implies that you will be pushing data to >>> the remote instance. >>> >>> Note: if the remote instance is configured to use secure data >>> transmission, you will see only ports that you are authorized to >>> communicate with. For information on configuring NiFi to run securely, see >>> the Admin Guide. >>> >>> In order to allow another NiFi instance to push data to your local >>> instance, you can simply drag an Input Port onto the Root Process Group of >>> your graph. After entering a name for the port, it will be added to your >>> flow. You can now right-click on the Input Port and choose Configure in >>> order to adjust the name and the number of concurrent tasks that are used >>> for the port. If Site-to-Site is configured to run securely, you will also >>> be given the ability to adjust who has access to the port. If secure, only >>> those who have been granted access to communicate with the port will be >>> able to see that the port exists. >>> >>> After being given access to a particular port, in order to see that port, >>> the operator of a remote NiFi instance may need to right-click on their >>> Remote Process Group and choose to "Refresh" the flow. >>> >>> Similar to an Input Port, a DataFlow Manager may choose to add an Output >>> Port to the Root Process Group. The Output Port allows an authorized NiFi >>> instance to remotely connect to your instance and pull data from the Output >>> Port. Configuring the Output Port will again allow the DFM to control how >>> many concurrent tasks are allowed, as well as which NiFi instances are >>> authorized to pull data from the instance being configured. >>> >>> In addition to other instances of NiFi, some other applications may use a >>> Site-to-Site client in order to push data to or receive data from a NiFi >>> instance. For example, NiFi provides an Apache Storm spout and an Apache >>> Spark Receiver that are able to pull data from NiFi’s Root Group Output >>> Ports. >>> >>> If your instance of NiFi is running securely, the first time that a client >>> establishes a connection to your instance, the client will be forbidden and >>> a request for an account for that client will automatically be generated. >>> The client will need to be granted the NiFi role in order to communicate >>> via Site-to-Site. For more information on managing user accounts, see >>> theControlling Levels of Access section of the Admin Guide. >>> >>> For information on how to enable and configure Site-to-Site on a NiFi >>> instance, see the Site-to-Site Properties section of the Admin Guide. >>> >>> Thanks >>> -Mark >>> >>> >>> ---------------------------------------- >>>> Date: Mon, 24 Aug 2015 19:36:55 -0400 >>>> Subject: NiFi Site-to-Site >>>> From: [email protected] >>>> To: [email protected] >>>> >>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with >>> Remote >>>> Process Groups? Any details on what port range(s) needs to be available? >>>> >>>> My setup: NiFi Producer provides data on output port "data-output" to be >>>> picked up by NiFi Consumer. >>>> >>>> While trying to get a simple setup running, I set >>>> "nifi.remote.input.secure" to false and "nifi.remote.input.socket.port" >>> to >>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The >>> Remote >>>> Process Group in NiFi Consumer has the "data-output" port turned on and >>> is >>>> simply routing to LogAttribute for verification. >>>> >>>> Logs in NiFi Producer indicate that it's periodically establishing a >>> socket >>>> connection with NiFi Consumer over ports in the 54000 - 55999 range, but >>> no >>>> data actually comes across to NiFi Consumer. I added opened NiFi >>>> Consumer's firewall for ingress from 54000 - 55999. >>>> >>>> Eventually, we'll need to get it over SSL and lock down the firewall to a >>>> minimum required range, but I'm hoping to prove it out first. >>>> >>>> Any guidance would be greatly appreciated! >>>> >>>> Thanks, >>>> Brian >>> >>> >> >> >> >> -- >> Brian Ghigiarelli >> 570-878-9139 >
