Hey Mark, Mixed results... Setting up an Output Port on the Producer and a Remote Process Group on the Receiver is still failing (with the exceptions from the previous email).
However, setting up an Input Port on the Receiver and a Remote Process Group on the Producer works just fine. Any ideas what could be going on there? Thanks, Brian On Thu, Aug 27, 2015 at 9:43 AM, Brian Ghigiarelli <[email protected]> wrote: > Hey Mark, > > I haven't had a chance to go back to verify, but my guess is that > specifying the nifi.remote.input.socket.host property will fix it. The > hostname on that box is not a valid hostname for the external NiFi that > needed to connect with it. For some reason, that property wasn't in the > nifi.properties file, so I didn't know that it was an available option. > Thanks for the pointers! > > Brian > > On Thu, Aug 27, 2015 at 8:44 AM, Mark Payne <[email protected]> wrote: > >> Hey Brian, >> >> Just wanted to follow up on this and see if you were able to get >> everything resolved, or if you >> are still running into problems. >> >> Thanks >> -Mark >> >> ---------------------------------------- >> > From: [email protected] >> > To: [email protected] >> > Subject: RE: NiFi Site-to-Site >> > Date: Tue, 25 Aug 2015 09:10:53 -0500 >> > >> > Brian, >> > >> > As part of the site-to-site communication, the client node (i.e., the >> NiFi node establishing the connection, >> > the one with the Remote Process Group) will connect and request a list >> of "peers" >> > (i.e., a list of nodes in the cluster). If you are running a very new >> version of 0.3.0-SNAPSHOT, you will >> > be able to configure what value is returned by that node by setting a >> value for the >> > "nifi.remote.input.socket.host" property in the nifi.properties file. >> If what you are running is a bit older, >> > or if that property is not set, what gets returned is the hostname that >> is returned by Java. >> > >> > Generally, when you see this error, it is because the hostname returned >> by the remote nifi instance >> > is not in your /etc/hosts file on the client. >> > >> > If you run the "hostname" command on the remote instance, can you >> verify that the value returned >> > is in /etc/hosts on the client instance? >> > >> > Thanks >> > -Mark >> > >> > ---------------------------------------- >> >> Date: Mon, 24 Aug 2015 21:41:19 -0400 >> >> Subject: Re: NiFi Site-to-Site >> >> From: [email protected] >> >> To: [email protected] >> >> >> >> Hey Mark, >> >> >> >> Thanks for the reply and the references. I've taken a look through, >> and it >> >> seems like everything should be set up properly. I am seeing the >> following >> >> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the >> >> NiFi Producer instance fine from the Consumer box, but Site-to-Site is >> >> still giving some trouble. That's why I started looking at AWS Security >> >> Groups and Firewall settings. Does this stack trace ring a bell with >> >> anything? >> >> >> >> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8] >> >> o.a.n.c.t.ContinuallyRunConnectableTask >> >> RemoteGroupPort[name=data-output,target= >> http://nifi-producer-host:8080/nifi] >> >> Administratively Pausing for 10 seconds due to processing failure: >> >> java.lang.RuntimeException: >> java.nio.channels.UnresolvedAddressException >> >> java.lang.RuntimeException: >> java.nio.channels.UnresolvedAddressException >> >> at >> >> >> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234) >> >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80) >> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40) >> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) >> >> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> [na:1.8.0_45] >> >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> >> [na:1.8.0_45] >> >> at >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> >> [na:1.8.0_45] >> >> at >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >> >> [na:1.8.0_45] >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> [na:1.8.0_45] >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> [na:1.8.0_45] >> >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] >> >> Caused by: java.nio.channels.UnresolvedAddressException: null >> >> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45] >> >> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) >> >> ~[na:1.8.0_45] >> >> at java.nio.channels.SocketChannel.open(SocketChannel.java:189) >> >> ~[na:1.8.0_45] >> >> at >> >> >> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708) >> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> >> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682) >> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> >> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300) >> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> >> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129) >> >> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> >> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167) >> >> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> at >> >> >> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227) >> >> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >> >> ... 10 common frames omitted >> >> >> >> Thanks again, >> >> Brian >> >> >> >> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <[email protected]> >> wrote: >> >> >> >>> Hey Brian, >> >>> >> >>> The latest version of the User Guide has instructions on setting up >> >>> site-to-site, but that version >> >>> hasn't yet been posted to the website, I don't believe. If you click >> the >> >>> "help" menu in the top-right >> >>> corner of your NiFi instance, and go to the User Guide from there, you >> >>> should have the up-to-date >> >>> guide there. >> >>> >> >>> If there's anything missing, please let us know so that we can update >> the >> >>> guide. >> >>> >> >>> For convenience, I have pasted the text of that section below: >> >>> >> >>> >> >>> In order to communicate with a remote NiFi instance via Site-to-Site, >> >>> simply drag a Remote Process Group onto the graph and enter the URL >> of the >> >>> remote NiFi instance (for more information on the components of a >> Remote >> >>> Process Group, see Remote Process Group Transmission section of this >> >>> guide.) The URL is the same URL you would use to go to that >> instance’s User >> >>> Interface. At that point, you can drag a connection to or from the >> Remote >> >>> Process Group in the same way you would drag a connection to or from a >> >>> Processor or a local Process Group. When you drag the connection, you >> will >> >>> have a chance to choose which Port to connect to. Note that it may >> take up >> >>> to one minute for the Remote Process Group to determine which ports >> are >> >>> available. >> >>> >> >>> If the connection is dragged starting from the Remote Process Group, >> the >> >>> ports shown will be the Output Ports of the remote group, as this >> indicates >> >>> that you will be pulling data from the remote instance. If the >> connection >> >>> instead ends on the Remote Process Group, the ports shown will be the >> Input >> >>> Ports of the remote group, as this implies that you will be pushing >> data to >> >>> the remote instance. >> >>> >> >>> Note: if the remote instance is configured to use secure data >> >>> transmission, you will see only ports that you are authorized to >> >>> communicate with. For information on configuring NiFi to run >> securely, see >> >>> the Admin Guide. >> >>> >> >>> In order to allow another NiFi instance to push data to your local >> >>> instance, you can simply drag an Input Port onto the Root Process >> Group of >> >>> your graph. After entering a name for the port, it will be added to >> your >> >>> flow. You can now right-click on the Input Port and choose Configure >> in >> >>> order to adjust the name and the number of concurrent tasks that are >> used >> >>> for the port. If Site-to-Site is configured to run securely, you will >> also >> >>> be given the ability to adjust who has access to the port. If secure, >> only >> >>> those who have been granted access to communicate with the port will >> be >> >>> able to see that the port exists. >> >>> >> >>> After being given access to a particular port, in order to see that >> port, >> >>> the operator of a remote NiFi instance may need to right-click on >> their >> >>> Remote Process Group and choose to "Refresh" the flow. >> >>> >> >>> Similar to an Input Port, a DataFlow Manager may choose to add an >> Output >> >>> Port to the Root Process Group. The Output Port allows an authorized >> NiFi >> >>> instance to remotely connect to your instance and pull data from the >> Output >> >>> Port. Configuring the Output Port will again allow the DFM to control >> how >> >>> many concurrent tasks are allowed, as well as which NiFi instances are >> >>> authorized to pull data from the instance being configured. >> >>> >> >>> In addition to other instances of NiFi, some other applications may >> use a >> >>> Site-to-Site client in order to push data to or receive data from a >> NiFi >> >>> instance. For example, NiFi provides an Apache Storm spout and an >> Apache >> >>> Spark Receiver that are able to pull data from NiFi’s Root Group >> Output >> >>> Ports. >> >>> >> >>> If your instance of NiFi is running securely, the first time that a >> client >> >>> establishes a connection to your instance, the client will be >> forbidden and >> >>> a request for an account for that client will automatically be >> generated. >> >>> The client will need to be granted the NiFi role in order to >> communicate >> >>> via Site-to-Site. For more information on managing user accounts, see >> >>> theControlling Levels of Access section of the Admin Guide. >> >>> >> >>> For information on how to enable and configure Site-to-Site on a NiFi >> >>> instance, see the Site-to-Site Properties section of the Admin Guide. >> >>> >> >>> Thanks >> >>> -Mark >> >>> >> >>> >> >>> ---------------------------------------- >> >>>> Date: Mon, 24 Aug 2015 19:36:55 -0400 >> >>>> Subject: NiFi Site-to-Site >> >>>> From: [email protected] >> >>>> To: [email protected] >> >>>> >> >>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with >> >>> Remote >> >>>> Process Groups? Any details on what port range(s) needs to be >> available? >> >>>> >> >>>> My setup: NiFi Producer provides data on output port "data-output" >> to be >> >>>> picked up by NiFi Consumer. >> >>>> >> >>>> While trying to get a simple setup running, I set >> >>>> "nifi.remote.input.secure" to false and >> "nifi.remote.input.socket.port" >> >>> to >> >>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The >> >>> Remote >> >>>> Process Group in NiFi Consumer has the "data-output" port turned on >> and >> >>> is >> >>>> simply routing to LogAttribute for verification. >> >>>> >> >>>> Logs in NiFi Producer indicate that it's periodically establishing a >> >>> socket >> >>>> connection with NiFi Consumer over ports in the 54000 - 55999 range, >> but >> >>> no >> >>>> data actually comes across to NiFi Consumer. I added opened NiFi >> >>>> Consumer's firewall for ingress from 54000 - 55999. >> >>>> >> >>>> Eventually, we'll need to get it over SSL and lock down the firewall >> to a >> >>>> minimum required range, but I'm hoping to prove it out first. >> >>>> >> >>>> Any guidance would be greatly appreciated! >> >>>> >> >>>> Thanks, >> >>>> Brian >> >>> >> >>> >> >> >> >> >> >> >> >> -- >> >> Brian Ghigiarelli >> >> 570-878-9139 >> > >> >> > > > > -- > Brian Ghigiarelli > 570-878-9139 > -- Brian Ghigiarelli 570-878-9139
