Hey Mark, I haven't had a chance to go back to verify, but my guess is that specifying the nifi.remote.input.socket.host property will fix it. The hostname on that box is not a valid hostname for the external NiFi that needed to connect with it. For some reason, that property wasn't in the nifi.properties file, so I didn't know that it was an available option. Thanks for the pointers!
Brian On Thu, Aug 27, 2015 at 8:44 AM, Mark Payne <[email protected]> wrote: > Hey Brian, > > Just wanted to follow up on this and see if you were able to get > everything resolved, or if you > are still running into problems. > > Thanks > -Mark > > ---------------------------------------- > > From: [email protected] > > To: [email protected] > > Subject: RE: NiFi Site-to-Site > > Date: Tue, 25 Aug 2015 09:10:53 -0500 > > > > Brian, > > > > As part of the site-to-site communication, the client node (i.e., the > NiFi node establishing the connection, > > the one with the Remote Process Group) will connect and request a list > of "peers" > > (i.e., a list of nodes in the cluster). If you are running a very new > version of 0.3.0-SNAPSHOT, you will > > be able to configure what value is returned by that node by setting a > value for the > > "nifi.remote.input.socket.host" property in the nifi.properties file. If > what you are running is a bit older, > > or if that property is not set, what gets returned is the hostname that > is returned by Java. > > > > Generally, when you see this error, it is because the hostname returned > by the remote nifi instance > > is not in your /etc/hosts file on the client. > > > > If you run the "hostname" command on the remote instance, can you verify > that the value returned > > is in /etc/hosts on the client instance? > > > > Thanks > > -Mark > > > > ---------------------------------------- > >> Date: Mon, 24 Aug 2015 21:41:19 -0400 > >> Subject: Re: NiFi Site-to-Site > >> From: [email protected] > >> To: [email protected] > >> > >> Hey Mark, > >> > >> Thanks for the reply and the references. I've taken a look through, and > it > >> seems like everything should be set up properly. I am seeing the > following > >> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the > >> NiFi Producer instance fine from the Consumer box, but Site-to-Site is > >> still giving some trouble. That's why I started looking at AWS Security > >> Groups and Firewall settings. Does this stack trace ring a bell with > >> anything? > >> > >> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8] > >> o.a.n.c.t.ContinuallyRunConnectableTask > >> RemoteGroupPort[name=data-output,target= > http://nifi-producer-host:8080/nifi] > >> Administratively Pausing for 10 seconds due to processing failure: > >> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException > >> java.lang.RuntimeException: java.nio.channels.UnresolvedAddressException > >> at > >> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234) > >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> > org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80) > >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> > org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40) > >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) > >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> [na:1.8.0_45] > >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > >> [na:1.8.0_45] > >> at > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > >> [na:1.8.0_45] > >> at > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > >> [na:1.8.0_45] > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > >> [na:1.8.0_45] > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > >> [na:1.8.0_45] > >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] > >> Caused by: java.nio.channels.UnresolvedAddressException: null > >> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45] > >> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) > >> ~[na:1.8.0_45] > >> at java.nio.channels.SocketChannel.open(SocketChannel.java:189) > >> ~[na:1.8.0_45] > >> at > >> > org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708) > >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> > org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682) > >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> > org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300) > >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> > org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129) > >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> > org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167) > >> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> at > >> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227) > >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > >> ... 10 common frames omitted > >> > >> Thanks again, > >> Brian > >> > >> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <[email protected]> > wrote: > >> > >>> Hey Brian, > >>> > >>> The latest version of the User Guide has instructions on setting up > >>> site-to-site, but that version > >>> hasn't yet been posted to the website, I don't believe. If you click > the > >>> "help" menu in the top-right > >>> corner of your NiFi instance, and go to the User Guide from there, you > >>> should have the up-to-date > >>> guide there. > >>> > >>> If there's anything missing, please let us know so that we can update > the > >>> guide. > >>> > >>> For convenience, I have pasted the text of that section below: > >>> > >>> > >>> In order to communicate with a remote NiFi instance via Site-to-Site, > >>> simply drag a Remote Process Group onto the graph and enter the URL of > the > >>> remote NiFi instance (for more information on the components of a > Remote > >>> Process Group, see Remote Process Group Transmission section of this > >>> guide.) The URL is the same URL you would use to go to that instance’s > User > >>> Interface. At that point, you can drag a connection to or from the > Remote > >>> Process Group in the same way you would drag a connection to or from a > >>> Processor or a local Process Group. When you drag the connection, you > will > >>> have a chance to choose which Port to connect to. Note that it may > take up > >>> to one minute for the Remote Process Group to determine which ports are > >>> available. > >>> > >>> If the connection is dragged starting from the Remote Process Group, > the > >>> ports shown will be the Output Ports of the remote group, as this > indicates > >>> that you will be pulling data from the remote instance. If the > connection > >>> instead ends on the Remote Process Group, the ports shown will be the > Input > >>> Ports of the remote group, as this implies that you will be pushing > data to > >>> the remote instance. > >>> > >>> Note: if the remote instance is configured to use secure data > >>> transmission, you will see only ports that you are authorized to > >>> communicate with. For information on configuring NiFi to run securely, > see > >>> the Admin Guide. > >>> > >>> In order to allow another NiFi instance to push data to your local > >>> instance, you can simply drag an Input Port onto the Root Process > Group of > >>> your graph. After entering a name for the port, it will be added to > your > >>> flow. You can now right-click on the Input Port and choose Configure in > >>> order to adjust the name and the number of concurrent tasks that are > used > >>> for the port. If Site-to-Site is configured to run securely, you will > also > >>> be given the ability to adjust who has access to the port. If secure, > only > >>> those who have been granted access to communicate with the port will be > >>> able to see that the port exists. > >>> > >>> After being given access to a particular port, in order to see that > port, > >>> the operator of a remote NiFi instance may need to right-click on their > >>> Remote Process Group and choose to "Refresh" the flow. > >>> > >>> Similar to an Input Port, a DataFlow Manager may choose to add an > Output > >>> Port to the Root Process Group. The Output Port allows an authorized > NiFi > >>> instance to remotely connect to your instance and pull data from the > Output > >>> Port. Configuring the Output Port will again allow the DFM to control > how > >>> many concurrent tasks are allowed, as well as which NiFi instances are > >>> authorized to pull data from the instance being configured. > >>> > >>> In addition to other instances of NiFi, some other applications may > use a > >>> Site-to-Site client in order to push data to or receive data from a > NiFi > >>> instance. For example, NiFi provides an Apache Storm spout and an > Apache > >>> Spark Receiver that are able to pull data from NiFi’s Root Group Output > >>> Ports. > >>> > >>> If your instance of NiFi is running securely, the first time that a > client > >>> establishes a connection to your instance, the client will be > forbidden and > >>> a request for an account for that client will automatically be > generated. > >>> The client will need to be granted the NiFi role in order to > communicate > >>> via Site-to-Site. For more information on managing user accounts, see > >>> theControlling Levels of Access section of the Admin Guide. > >>> > >>> For information on how to enable and configure Site-to-Site on a NiFi > >>> instance, see the Site-to-Site Properties section of the Admin Guide. > >>> > >>> Thanks > >>> -Mark > >>> > >>> > >>> ---------------------------------------- > >>>> Date: Mon, 24 Aug 2015 19:36:55 -0400 > >>>> Subject: NiFi Site-to-Site > >>>> From: [email protected] > >>>> To: [email protected] > >>>> > >>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with > >>> Remote > >>>> Process Groups? Any details on what port range(s) needs to be > available? > >>>> > >>>> My setup: NiFi Producer provides data on output port "data-output" to > be > >>>> picked up by NiFi Consumer. > >>>> > >>>> While trying to get a simple setup running, I set > >>>> "nifi.remote.input.secure" to false and > "nifi.remote.input.socket.port" > >>> to > >>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The > >>> Remote > >>>> Process Group in NiFi Consumer has the "data-output" port turned on > and > >>> is > >>>> simply routing to LogAttribute for verification. > >>>> > >>>> Logs in NiFi Producer indicate that it's periodically establishing a > >>> socket > >>>> connection with NiFi Consumer over ports in the 54000 - 55999 range, > but > >>> no > >>>> data actually comes across to NiFi Consumer. I added opened NiFi > >>>> Consumer's firewall for ingress from 54000 - 55999. > >>>> > >>>> Eventually, we'll need to get it over SSL and lock down the firewall > to a > >>>> minimum required range, but I'm hoping to prove it out first. > >>>> > >>>> Any guidance would be greatly appreciated! > >>>> > >>>> Thanks, > >>>> Brian > >>> > >>> > >> > >> > >> > >> -- > >> Brian Ghigiarelli > >> 570-878-9139 > > > > -- Brian Ghigiarelli 570-878-9139
