Hey Mark, Thanks for the reply and the references. I've taken a look through, and it seems like everything should be set up properly. I am seeing the following exception in the NiFi Consumer nifi-app.log. I can ping and wget to the NiFi Producer instance fine from the Consumer box, but Site-to-Site is still giving some trouble. That's why I started looking at AWS Security Groups and Firewall settings. Does this stack trace ring a bell with anything?
2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunConnectableTask RemoteGroupPort[name=data-output,target=http://nifi-producer-host:8080/nifi] Administratively Pausing for 10 seconds due to processing failure: java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException at org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234) ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80) [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40) [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] Caused by: java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_45] at java.nio.channels.SocketChannel.open(SocketChannel.java:189) ~[na:1.8.0_45] at org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708) ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682) ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300) ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129) ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167) ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227) ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] ... 10 common frames omitted Thanks again, Brian On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <[email protected]> wrote: > Hey Brian, > > The latest version of the User Guide has instructions on setting up > site-to-site, but that version > hasn't yet been posted to the website, I don't believe. If you click the > "help" menu in the top-right > corner of your NiFi instance, and go to the User Guide from there, you > should have the up-to-date > guide there. > > If there's anything missing, please let us know so that we can update the > guide. > > For convenience, I have pasted the text of that section below: > > > In order to communicate with a remote NiFi instance via Site-to-Site, > simply drag a Remote Process Group onto the graph and enter the URL of the > remote NiFi instance (for more information on the components of a Remote > Process Group, see Remote Process Group Transmission section of this > guide.) The URL is the same URL you would use to go to that instance’s User > Interface. At that point, you can drag a connection to or from the Remote > Process Group in the same way you would drag a connection to or from a > Processor or a local Process Group. When you drag the connection, you will > have a chance to choose which Port to connect to. Note that it may take up > to one minute for the Remote Process Group to determine which ports are > available. > > If the connection is dragged starting from the Remote Process Group, the > ports shown will be the Output Ports of the remote group, as this indicates > that you will be pulling data from the remote instance. If the connection > instead ends on the Remote Process Group, the ports shown will be the Input > Ports of the remote group, as this implies that you will be pushing data to > the remote instance. > > Note: if the remote instance is configured to use secure data > transmission, you will see only ports that you are authorized to > communicate with. For information on configuring NiFi to run securely, see > the Admin Guide. > > In order to allow another NiFi instance to push data to your local > instance, you can simply drag an Input Port onto the Root Process Group of > your graph. After entering a name for the port, it will be added to your > flow. You can now right-click on the Input Port and choose Configure in > order to adjust the name and the number of concurrent tasks that are used > for the port. If Site-to-Site is configured to run securely, you will also > be given the ability to adjust who has access to the port. If secure, only > those who have been granted access to communicate with the port will be > able to see that the port exists. > > After being given access to a particular port, in order to see that port, > the operator of a remote NiFi instance may need to right-click on their > Remote Process Group and choose to "Refresh" the flow. > > Similar to an Input Port, a DataFlow Manager may choose to add an Output > Port to the Root Process Group. The Output Port allows an authorized NiFi > instance to remotely connect to your instance and pull data from the Output > Port. Configuring the Output Port will again allow the DFM to control how > many concurrent tasks are allowed, as well as which NiFi instances are > authorized to pull data from the instance being configured. > > In addition to other instances of NiFi, some other applications may use a > Site-to-Site client in order to push data to or receive data from a NiFi > instance. For example, NiFi provides an Apache Storm spout and an Apache > Spark Receiver that are able to pull data from NiFi’s Root Group Output > Ports. > > If your instance of NiFi is running securely, the first time that a client > establishes a connection to your instance, the client will be forbidden and > a request for an account for that client will automatically be generated. > The client will need to be granted the NiFi role in order to communicate > via Site-to-Site. For more information on managing user accounts, see > theControlling Levels of Access section of the Admin Guide. > > For information on how to enable and configure Site-to-Site on a NiFi > instance, see the Site-to-Site Properties section of the Admin Guide. > > Thanks > -Mark > > > ---------------------------------------- > > Date: Mon, 24 Aug 2015 19:36:55 -0400 > > Subject: NiFi Site-to-Site > > From: [email protected] > > To: [email protected] > > > > Is there any step-by-step guide to setting up NiFi Site-to-Site with > Remote > > Process Groups? Any details on what port range(s) needs to be available? > > > > My setup: NiFi Producer provides data on output port "data-output" to be > > picked up by NiFi Consumer. > > > > While trying to get a simple setup running, I set > > "nifi.remote.input.secure" to false and "nifi.remote.input.socket.port" > to > > 8082 in NiFi Producer (also added TCP ingress to the firewall). The > Remote > > Process Group in NiFi Consumer has the "data-output" port turned on and > is > > simply routing to LogAttribute for verification. > > > > Logs in NiFi Producer indicate that it's periodically establishing a > socket > > connection with NiFi Consumer over ports in the 54000 - 55999 range, but > no > > data actually comes across to NiFi Consumer. I added opened NiFi > > Consumer's firewall for ingress from 54000 - 55999. > > > > Eventually, we'll need to get it over SSL and lock down the firewall to a > > minimum required range, but I'm hoping to prove it out first. > > > > Any guidance would be greatly appreciated! > > > > Thanks, > > Brian > > -- Brian Ghigiarelli 570-878-9139
