That long thread has no attached images since LifeLock seems unable to receive them.
On Thu, Jan 21, 2016 at 2:48 PM, Joe Witt <[email protected]> wrote: > not to take away from the thread but Matt I didn't get your images either > > On Thu, Jan 21, 2016 at 2:45 PM, Matthew Clarke > <[email protected]> wrote: > > Paresh, > > > > I am really not sure why you are not getting by attachments. Other > > are receiving them fine. Is it possible the LifeLock email servers are > > dropping them? > > > > I think we need to take one step back and talk a little more about > > what input and output ports are used for. > > > > Lets start by talking about the NiFi canvas where you are creating > > your dataflows. When you started NiFi for the very first time you were > > given a blank canvas. You can think of this canvas as the root process > > group. From there you were able to add additional process groups to that > > top level canvas. These added process groups allowed you drill down in to > > them giving additional blank canvases you could build dataflows on. When > > you enter an added process group you will see the hierarchy represented > > just above the canvas in the UI ( flowname >> processgroupname ). NiFi > > does not restrict the number of process groups your create or the depth > you > > can go with them. You could compare the process group hierarchy to that > of > > a Windows directory structure. So if i added another process group inside > > one tat i already created, i have essentially now gone two layers deeper. > > ( flowname >> processgroupname >> processgroupdeeper ). The hierarchy > > represented above you canvas allows you to quickly jump up one or more > > layers all the way to the root level. > > > > Now that we understand process groups, lets talk about how we move > > data in and out of these process groups. This is where input and output > > ports come in to play. Input and output ports exist to move FlowFIles > > between a process group and *ONE LEVEL UP *from that process group. Input > > ports will accept FlowFiles coming from one level up and output ports > allow > > FlowFiles to be sent one level up. If I have a process group added to my > > canvas, I cannot drag a connection to it until at least one input port > > exists inside that process group. I also cannot drag a connection off of > > that process group until at least on output port exists inside the > process > > group. You can only move FlowFiles up or down one level at a time. Given > > the example of a process group within another process group, FlowFiles > > would need to be moved from the deepest level up to the middle layer > before > > finally being able to be moved to the root canvas. > > > > Site-to-Site behaves and is configured very much in the same way. > > Instead of moving FlowFiles between different process groups (layers) > > within the same NiFi, we are moving FlowFiles between different NiFi > > instances or clusters. As I explained earlier, input and output ports > are > > used to move data to and from one level up. At the top level of your > > canvas (root process group level) adding input or output ports provides > the > > ability for that NiFi to receive (input port) FlowFiles from another NiFi > > instance or have a another NiFI pull files from (output port) that NiFi. > > We refer to this ability to send FlowFiles between different NiFi > instances > > as Site-to-Site. We refer to input and output ports added the top level > as > > remote input or output ports. These Remote Process group are not > configured > > with unique system port numbers, but instead all utilize the same > > Site-To-Site port number configured in your nifi.properties files. They > > are added the same way as you would add any other input port they just > look > > a little different when added at the top level. Their configuration has > two > > tabs instead of one. Now that you have your receiving NiFi configured > with > > input ports and/or output ports. you need to configure another NiFi to > send > > FlowFile to or pull FlowFiles from these ports. > > > > In single instance you can send data to an input port inside a > > process group by dragging a connection to the process group and selecting > > the name of the input port from a selection menu provided. However on > your > > source NiFi you need to add a Remote Process Group (RPG) that uses the > URL > > for the target NiFi (NCM URL if target is a NiFI cluster). Once the > > initial communications has occurred, you can drag a connection to the RPG > > much in the same way you previously moved FlowFiles between process > groups > > (layers) within a single instance of NiFi. You can also hover over the > > RPG and drag a connection off of the RPG which will allow you to pull > data > > from an available output port on the target NiFi. > > > > Remember that the Source NiFi (standalone or cluster) has one or > more > > RPGs while the target NiFi contains the input and output ports (Input > ports > > added to lop level). In your setup it sounds like all your dataflows on > > the target NiFis exist inside process groups. So you would need to take > > data received on your top level input ports and connect it into your > > process groups. > > > > Thanks, > > Matt > > > > > > > > On Wed, Jan 20, 2016 at 9:11 PM, Paresh Shah <[email protected]> > > wrote: > > > >> Another question. > >> > >> To use the Remote Processor Group, on the sender side should we do the > >> following. > >> > >> 1. Create Output Port. > >> 2. Create Remote Processor Group using the OutputPort created. > >> > >> Or > >> > >> 1. Create the target pipeline with the input port > >> 2. Create Remote Processor Group with the input port -> This > gives > >> the error that it has timed out reading from the port. But we need to be > >> sending the data to this port rather then read from it. > >> > >> Thanks > >> Paresh > >> > >> From: Paresh Shah <[email protected]<mailto: > >> [email protected]>> > >> Date: Wednesday, January 20, 2016 at 4:01 PM > >> To: "[email protected]<mailto:[email protected]>" < > [email protected] > >> <mailto:[email protected]>> > >> Subject: Re: How to configure/start multiple Input ports. > >> > >> Still don’t see the images. > >> > >> Also if the Input Ports are at the root canvas, we are not able to > connect > >> it to the ProcessorGroup. We have all our pipelines within different > >> processor groups. > >> > >> Paresh > >> > >> From: Matthew Clarke <[email protected]<mailto: > >> [email protected]>> > >> Reply-To: "[email protected]<mailto:[email protected]>" < > >> [email protected]<mailto:[email protected]>> > >> Date: Wednesday, January 20, 2016 at 2:37 PM > >> To: "[email protected]<mailto:[email protected]>" < > [email protected] > >> <mailto:[email protected]>> > >> Subject: Re: How to configure/start multiple Input ports. > >> > >> Three images attached: > >> > >> On Wed, Jan 20, 2016 at 4:31 PM, Paresh Shah <[email protected] > >> <mailto:[email protected]>> wrote: > >> I cannot see the pictures. Can you please resend it with attachements. > >> > >> Thanks > >> Paresh > >> > >> From: Matthew Clarke <[email protected]<mailto: > >> [email protected]><mailto:[email protected]<mailto: > >> [email protected]>>> > >> Reply-To: "[email protected]<mailto:[email protected]><mailto: > >> [email protected]<mailto:[email protected]>>" <[email protected] > >> <mailto:[email protected]><mailto:[email protected]<mailto: > >> [email protected]>>> > >> Date: Wednesday, January 20, 2016 at 12:55 PM > >> To: "[email protected]<mailto:[email protected]><mailto: > >> [email protected]<mailto:[email protected]>>" <[email protected] > >> <mailto:[email protected]><mailto:[email protected]<mailto: > >> [email protected]>>> > >> Subject: Re: How to configure/start multiple Input ports. > >> > >> Paresh, > >> All Site-to-Site data comes across a single port connection. > When > >> adding a RPG to the source NiFi, the URL you provide will be the URL for > >> the NCM of your internal NiFi cluster. You will add a a different RPG > for > >> each cluster you are connecting to. so in your case the external facing > >> NiFi cluster will have 2 RPGs on the graph with 1 configured form the > URL > >> of the NCM on internal NiFi cluster 1 and the other for the NCM of > internal > >> NiFi cluster 2. Once a connection is established to an NCM, the NCM > will > >> communicate to the RPG the URLs and site-to-site ports for each of the > >> clusters connected Nodes as well as the current load on those nodes. > The > >> NiFi with the RPG will then do a smart load-balanced delivery of data to > >> those nodes. On the target cluster you will add input ports at the root > >> level of the graph (They cannot be nested inside of any process groups) > . > >> You can add as many uniquely named input ports as you would like. These > >> input ports will be exposed to the RPG on the sending system. When you > >> draw a connection to the RPG, you will be provided witha pull-down > >> selection menu of all available ports. Selecting one will complete this > >> connection. So all data between NiFi systems via Site-to-Site will use > the > >> same site-to-site port but as you can see it enters the flow on the > >> destination system via input ports. > >> > >> [Inline image 1] > >> > >> So on your externally visible NiFi you would have two RPGs setup like > >> below: > >> [Inline image 3] > >> > >> The receiving NiFi cluster will have one or more uniquely named input > >> ports: > >> > >> [Inline image 4] > >> > >> As you can see from the above two screenshots the sending system's > >> connection to the RPG shows that the connection as being connected to > the > >> input port "Node-Input" on the receiving cluster. > >> > >> > >> Thanks, > >> Matt > >> > >> > >> > >> On Wed, Jan 20, 2016 at 2:59 PM, Paresh Shah <[email protected] > >> <mailto:[email protected]><mailto:[email protected] > <mailto: > >> [email protected]>>> wrote: > >> > >> We are trying to see how to run our pipelines in a clustered env using > >> site-to-site. Our scenario is something like the following. > >> > >> 1. 3 different clusters. > >> 2. One of the clusters is externally visible and will be the primary > >> cluster. > >> 3. All the data receivers will run on the primary node. > >> > >> Each of the pipelines has a RemoteProcessorGroup( RPG ) that would in > >> –turn forward the data to their corresponding Input port. > >> > >> Looking at the way the site-to-site configuration indicates that we are > >> only able to specify one port that would be started on a particular > node. > >> So the question is how can we pin a particular RPG to it corresponding > >> InputPort. > >> > >> Any insights would be greatly appreciated. > >> > >> Thanks > >> Paresh > >> ________________________________ > >> The information contained in this transmission may contain privileged > and > >> confidential information. It is intended only for the use of the > person(s) > >> named above. If you are not the intended recipient, you are hereby > notified > >> that any review, dissemination, distribution or duplication of this > >> communication is strictly prohibited. If you are not the intended > >> recipient, please contact the sender by reply email and destroy all > copies > >> of the original message. > >> ________________________________ > >> > >> ________________________________ > >> The information contained in this transmission may contain privileged > and > >> confidential information. It is intended only for the use of the > person(s) > >> named above. If you are not the intended recipient, you are hereby > notified > >> that any review, dissemination, distribution or duplication of this > >> communication is strictly prohibited. If you are not the intended > >> recipient, please contact the sender by reply email and destroy all > copies > >> of the original message. > >> ________________________________ > >> > >> ________________________________ > >> The information contained in this transmission may contain privileged > and > >> confidential information. It is intended only for the use of the > person(s) > >> named above. If you are not the intended recipient, you are hereby > notified > >> that any review, dissemination, distribution or duplication of this > >> communication is strictly prohibited. If you are not the intended > >> recipient, please contact the sender by reply email and destroy all > copies > >> of the original message. > >> ________________________________ > >> >
