Brian, The node that has the Remote Process Group is the "client". The one that has the port is the one that returns the hostname. So I believe what is happening is that when you are pulling data, the node that is trying to pull does not recognize the hostname returned by the sender. When you are pushing data, the host doing the pushes does recognize the hostname returned by the receiver.
Also, if you use the "nifi.remote.input.socket.host" property, that is only used right now when connecting to a standalone node. There is a ticket [1] to make it work in a clustered environment. [1] https://issues.apache.org/jira/browse/NIFI-872 ---------------------------------------- > Date: Thu, 27 Aug 2015 14:40:58 -0400 > Subject: Re: NiFi Site-to-Site > From: [email protected] > To: [email protected] > > Hey Mark, > > Mixed results... Setting up an Output Port on the Producer and a Remote > Process Group on the Receiver is still failing (with the exceptions from > the previous email). > > However, setting up an Input Port on the Receiver and a Remote Process > Group on the Producer works just fine. > > Any ideas what could be going on there? > > Thanks, > Brian > > On Thu, Aug 27, 2015 at 9:43 AM, Brian Ghigiarelli <[email protected]> > wrote: > >> Hey Mark, >> >> I haven't had a chance to go back to verify, but my guess is that >> specifying the nifi.remote.input.socket.host property will fix it. The >> hostname on that box is not a valid hostname for the external NiFi that >> needed to connect with it. For some reason, that property wasn't in the >> nifi.properties file, so I didn't know that it was an available option. >> Thanks for the pointers! >> >> Brian >> >> On Thu, Aug 27, 2015 at 8:44 AM, Mark Payne <[email protected]> wrote: >> >>> Hey Brian, >>> >>> Just wanted to follow up on this and see if you were able to get >>> everything resolved, or if you >>> are still running into problems. >>> >>> Thanks >>> -Mark >>> >>> ---------------------------------------- >>>> From: [email protected] >>>> To: [email protected] >>>> Subject: RE: NiFi Site-to-Site >>>> Date: Tue, 25 Aug 2015 09:10:53 -0500 >>>> >>>> Brian, >>>> >>>> As part of the site-to-site communication, the client node (i.e., the >>> NiFi node establishing the connection, >>>> the one with the Remote Process Group) will connect and request a list >>> of "peers" >>>> (i.e., a list of nodes in the cluster). If you are running a very new >>> version of 0.3.0-SNAPSHOT, you will >>>> be able to configure what value is returned by that node by setting a >>> value for the >>>> "nifi.remote.input.socket.host" property in the nifi.properties file. >>> If what you are running is a bit older, >>>> or if that property is not set, what gets returned is the hostname that >>> is returned by Java. >>>> >>>> Generally, when you see this error, it is because the hostname returned >>> by the remote nifi instance >>>> is not in your /etc/hosts file on the client. >>>> >>>> If you run the "hostname" command on the remote instance, can you >>> verify that the value returned >>>> is in /etc/hosts on the client instance? >>>> >>>> Thanks >>>> -Mark >>>> >>>> ---------------------------------------- >>>>> Date: Mon, 24 Aug 2015 21:41:19 -0400 >>>>> Subject: Re: NiFi Site-to-Site >>>>> From: [email protected] >>>>> To: [email protected] >>>>> >>>>> Hey Mark, >>>>> >>>>> Thanks for the reply and the references. I've taken a look through, >>> and it >>>>> seems like everything should be set up properly. I am seeing the >>> following >>>>> exception in the NiFi Consumer nifi-app.log. I can ping and wget to the >>>>> NiFi Producer instance fine from the Consumer box, but Site-to-Site is >>>>> still giving some trouble. That's why I started looking at AWS Security >>>>> Groups and Firewall settings. Does this stack trace ring a bell with >>>>> anything? >>>>> >>>>> 2015-08-25 01:34:30,709 WARN [Timer-Driven Process Thread-8] >>>>> o.a.n.c.t.ContinuallyRunConnectableTask >>>>> RemoteGroupPort[name=data-output,target= >>> http://nifi-producer-host:8080/nifi] >>>>> Administratively Pausing for 10 seconds due to processing failure: >>>>> java.lang.RuntimeException: >>> java.nio.channels.UnresolvedAddressException >>>>> java.lang.RuntimeException: >>> java.nio.channels.UnresolvedAddressException >>>>> at >>>>> >>> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:234) >>>>> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> >>> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80) >>>>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> >>> org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40) >>>>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> >>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) >>>>> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>> [na:1.8.0_45] >>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >>>>> [na:1.8.0_45] >>>>> at >>>>> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >>>>> [na:1.8.0_45] >>>>> at >>>>> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >>>>> [na:1.8.0_45] >>>>> at >>>>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> [na:1.8.0_45] >>>>> at >>>>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> [na:1.8.0_45] >>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] >>>>> Caused by: java.nio.channels.UnresolvedAddressException: null >>>>> at sun.nio.ch.Net.checkAddress(Net.java:123) ~[na:1.8.0_45] >>>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) >>>>> ~[na:1.8.0_45] >>>>> at java.nio.channels.SocketChannel.open(SocketChannel.java:189) >>>>> ~[na:1.8.0_45] >>>>> at >>>>> >>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708) >>>>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> >>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:682) >>>>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> >>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:300) >>>>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> >>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129) >>>>> ~[nifi-site-to-site-client-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> >>> org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:167) >>>>> ~[nifi-site-to-site-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> at >>>>> >>> org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:227) >>>>> ~[nifi-framework-core-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] >>>>> ... 10 common frames omitted >>>>> >>>>> Thanks again, >>>>> Brian >>>>> >>>>> On Mon, Aug 24, 2015 at 8:05 PM, Mark Payne <[email protected]> >>> wrote: >>>>> >>>>>> Hey Brian, >>>>>> >>>>>> The latest version of the User Guide has instructions on setting up >>>>>> site-to-site, but that version >>>>>> hasn't yet been posted to the website, I don't believe. If you click >>> the >>>>>> "help" menu in the top-right >>>>>> corner of your NiFi instance, and go to the User Guide from there, you >>>>>> should have the up-to-date >>>>>> guide there. >>>>>> >>>>>> If there's anything missing, please let us know so that we can update >>> the >>>>>> guide. >>>>>> >>>>>> For convenience, I have pasted the text of that section below: >>>>>> >>>>>> >>>>>> In order to communicate with a remote NiFi instance via Site-to-Site, >>>>>> simply drag a Remote Process Group onto the graph and enter the URL >>> of the >>>>>> remote NiFi instance (for more information on the components of a >>> Remote >>>>>> Process Group, see Remote Process Group Transmission section of this >>>>>> guide.) The URL is the same URL you would use to go to that >>> instance’s User >>>>>> Interface. At that point, you can drag a connection to or from the >>> Remote >>>>>> Process Group in the same way you would drag a connection to or from a >>>>>> Processor or a local Process Group. When you drag the connection, you >>> will >>>>>> have a chance to choose which Port to connect to. Note that it may >>> take up >>>>>> to one minute for the Remote Process Group to determine which ports >>> are >>>>>> available. >>>>>> >>>>>> If the connection is dragged starting from the Remote Process Group, >>> the >>>>>> ports shown will be the Output Ports of the remote group, as this >>> indicates >>>>>> that you will be pulling data from the remote instance. If the >>> connection >>>>>> instead ends on the Remote Process Group, the ports shown will be the >>> Input >>>>>> Ports of the remote group, as this implies that you will be pushing >>> data to >>>>>> the remote instance. >>>>>> >>>>>> Note: if the remote instance is configured to use secure data >>>>>> transmission, you will see only ports that you are authorized to >>>>>> communicate with. For information on configuring NiFi to run >>> securely, see >>>>>> the Admin Guide. >>>>>> >>>>>> In order to allow another NiFi instance to push data to your local >>>>>> instance, you can simply drag an Input Port onto the Root Process >>> Group of >>>>>> your graph. After entering a name for the port, it will be added to >>> your >>>>>> flow. You can now right-click on the Input Port and choose Configure >>> in >>>>>> order to adjust the name and the number of concurrent tasks that are >>> used >>>>>> for the port. If Site-to-Site is configured to run securely, you will >>> also >>>>>> be given the ability to adjust who has access to the port. If secure, >>> only >>>>>> those who have been granted access to communicate with the port will >>> be >>>>>> able to see that the port exists. >>>>>> >>>>>> After being given access to a particular port, in order to see that >>> port, >>>>>> the operator of a remote NiFi instance may need to right-click on >>> their >>>>>> Remote Process Group and choose to "Refresh" the flow. >>>>>> >>>>>> Similar to an Input Port, a DataFlow Manager may choose to add an >>> Output >>>>>> Port to the Root Process Group. The Output Port allows an authorized >>> NiFi >>>>>> instance to remotely connect to your instance and pull data from the >>> Output >>>>>> Port. Configuring the Output Port will again allow the DFM to control >>> how >>>>>> many concurrent tasks are allowed, as well as which NiFi instances are >>>>>> authorized to pull data from the instance being configured. >>>>>> >>>>>> In addition to other instances of NiFi, some other applications may >>> use a >>>>>> Site-to-Site client in order to push data to or receive data from a >>> NiFi >>>>>> instance. For example, NiFi provides an Apache Storm spout and an >>> Apache >>>>>> Spark Receiver that are able to pull data from NiFi’s Root Group >>> Output >>>>>> Ports. >>>>>> >>>>>> If your instance of NiFi is running securely, the first time that a >>> client >>>>>> establishes a connection to your instance, the client will be >>> forbidden and >>>>>> a request for an account for that client will automatically be >>> generated. >>>>>> The client will need to be granted the NiFi role in order to >>> communicate >>>>>> via Site-to-Site. For more information on managing user accounts, see >>>>>> theControlling Levels of Access section of the Admin Guide. >>>>>> >>>>>> For information on how to enable and configure Site-to-Site on a NiFi >>>>>> instance, see the Site-to-Site Properties section of the Admin Guide. >>>>>> >>>>>> Thanks >>>>>> -Mark >>>>>> >>>>>> >>>>>> ---------------------------------------- >>>>>>> Date: Mon, 24 Aug 2015 19:36:55 -0400 >>>>>>> Subject: NiFi Site-to-Site >>>>>>> From: [email protected] >>>>>>> To: [email protected] >>>>>>> >>>>>>> Is there any step-by-step guide to setting up NiFi Site-to-Site with >>>>>> Remote >>>>>>> Process Groups? Any details on what port range(s) needs to be >>> available? >>>>>>> >>>>>>> My setup: NiFi Producer provides data on output port "data-output" >>> to be >>>>>>> picked up by NiFi Consumer. >>>>>>> >>>>>>> While trying to get a simple setup running, I set >>>>>>> "nifi.remote.input.secure" to false and >>> "nifi.remote.input.socket.port" >>>>>> to >>>>>>> 8082 in NiFi Producer (also added TCP ingress to the firewall). The >>>>>> Remote >>>>>>> Process Group in NiFi Consumer has the "data-output" port turned on >>> and >>>>>> is >>>>>>> simply routing to LogAttribute for verification. >>>>>>> >>>>>>> Logs in NiFi Producer indicate that it's periodically establishing a >>>>>> socket >>>>>>> connection with NiFi Consumer over ports in the 54000 - 55999 range, >>> but >>>>>> no >>>>>>> data actually comes across to NiFi Consumer. I added opened NiFi >>>>>>> Consumer's firewall for ingress from 54000 - 55999. >>>>>>> >>>>>>> Eventually, we'll need to get it over SSL and lock down the firewall >>> to a >>>>>>> minimum required range, but I'm hoping to prove it out first. >>>>>>> >>>>>>> Any guidance would be greatly appreciated! >>>>>>> >>>>>>> Thanks, >>>>>>> Brian >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Brian Ghigiarelli >>>>> 570-878-9139 >>>> >>> >>> >> >> >> >> -- >> Brian Ghigiarelli >> 570-878-9139 >> > > > > -- > Brian Ghigiarelli > 570-878-9139
