Paresh,
I am really not sure why you are not getting by attachments. Other
are receiving them fine. Is it possible the LifeLock email servers are
dropping them?
I think we need to take one step back and talk a little more about
what input and output ports are used for.
Lets start by talking about the NiFi canvas where you are creating
your dataflows. When you started NiFi for the very first time you were
given a blank canvas. You can think of this canvas as the root process
group. From there you were able to add additional process groups to that
top level canvas. These added process groups allowed you drill down in to
them giving additional blank canvases you could build dataflows on. When
you enter an added process group you will see the hierarchy represented
just above the canvas in the UI ( flowname >> processgroupname ). NiFi
does not restrict the number of process groups your create or the depth you
can go with them. You could compare the process group hierarchy to that of
a Windows directory structure. So if i added another process group inside
one tat i already created, i have essentially now gone two layers deeper.
( flowname >> processgroupname >> processgroupdeeper ). The hierarchy
represented above you canvas allows you to quickly jump up one or more
layers all the way to the root level.
Now that we understand process groups, lets talk about how we move
data in and out of these process groups. This is where input and output
ports come in to play. Input and output ports exist to move FlowFIles
between a process group and *ONE LEVEL UP *from that process group. Input
ports will accept FlowFiles coming from one level up and output ports allow
FlowFiles to be sent one level up. If I have a process group added to my
canvas, I cannot drag a connection to it until at least one input port
exists inside that process group. I also cannot drag a connection off of
that process group until at least on output port exists inside the process
group. You can only move FlowFiles up or down one level at a time. Given
the example of a process group within another process group, FlowFiles
would need to be moved from the deepest level up to the middle layer before
finally being able to be moved to the root canvas.
Site-to-Site behaves and is configured very much in the same way.
Instead of moving FlowFiles between different process groups (layers)
within the same NiFi, we are moving FlowFiles between different NiFi
instances or clusters. As I explained earlier, input and output ports are
used to move data to and from one level up. At the top level of your
canvas (root process group level) adding input or output ports provides the
ability for that NiFi to receive (input port) FlowFiles from another NiFi
instance or have a another NiFI pull files from (output port) that NiFi.
We refer to this ability to send FlowFiles between different NiFi instances
as Site-to-Site. We refer to input and output ports added the top level as
remote input or output ports. These Remote Process group are not configured
with unique system port numbers, but instead all utilize the same
Site-To-Site port number configured in your nifi.properties files. They
are added the same way as you would add any other input port they just look
a little different when added at the top level. Their configuration has two
tabs instead of one. Now that you have your receiving NiFi configured with
input ports and/or output ports. you need to configure another NiFi to send
FlowFile to or pull FlowFiles from these ports.
In single instance you can send data to an input port inside a
process group by dragging a connection to the process group and selecting
the name of the input port from a selection menu provided. However on your
source NiFi you need to add a Remote Process Group (RPG) that uses the URL
for the target NiFi (NCM URL if target is a NiFI cluster). Once the
initial communications has occurred, you can drag a connection to the RPG
much in the same way you previously moved FlowFiles between process groups
(layers) within a single instance of NiFi. You can also hover over the
RPG and drag a connection off of the RPG which will allow you to pull data
from an available output port on the target NiFi.
Remember that the Source NiFi (standalone or cluster) has one or more
RPGs while the target NiFi contains the input and output ports (Input ports
added to lop level). In your setup it sounds like all your dataflows on
the target NiFis exist inside process groups. So you would need to take
data received on your top level input ports and connect it into your
process groups.
Thanks,
Matt
On Wed, Jan 20, 2016 at 9:11 PM, Paresh Shah <[email protected]>
wrote:
> Another question.
>
> To use the Remote Processor Group, on the sender side should we do the
> following.
>
> 1. Create Output Port.
> 2. Create Remote Processor Group using the OutputPort created.
>
> Or
>
> 1. Create the target pipeline with the input port
> 2. Create Remote Processor Group with the input port -> This gives
> the error that it has timed out reading from the port. But we need to be
> sending the data to this port rather then read from it.
>
> Thanks
> Paresh
>
> From: Paresh Shah <[email protected]<mailto:
> [email protected]>>
> Date: Wednesday, January 20, 2016 at 4:01 PM
> To: "[email protected]<mailto:[email protected]>" <[email protected]
> <mailto:[email protected]>>
> Subject: Re: How to configure/start multiple Input ports.
>
> Still don’t see the images.
>
> Also if the Input Ports are at the root canvas, we are not able to connect
> it to the ProcessorGroup. We have all our pipelines within different
> processor groups.
>
> Paresh
>
> From: Matthew Clarke <[email protected]<mailto:
> [email protected]>>
> Reply-To: "[email protected]<mailto:[email protected]>" <
> [email protected]<mailto:[email protected]>>
> Date: Wednesday, January 20, 2016 at 2:37 PM
> To: "[email protected]<mailto:[email protected]>" <[email protected]
> <mailto:[email protected]>>
> Subject: Re: How to configure/start multiple Input ports.
>
> Three images attached:
>
> On Wed, Jan 20, 2016 at 4:31 PM, Paresh Shah <[email protected]
> <mailto:[email protected]>> wrote:
> I cannot see the pictures. Can you please resend it with attachements.
>
> Thanks
> Paresh
>
> From: Matthew Clarke <[email protected]<mailto:
> [email protected]><mailto:[email protected]<mailto:
> [email protected]>>>
> Reply-To: "[email protected]<mailto:[email protected]><mailto:
> [email protected]<mailto:[email protected]>>" <[email protected]
> <mailto:[email protected]><mailto:[email protected]<mailto:
> [email protected]>>>
> Date: Wednesday, January 20, 2016 at 12:55 PM
> To: "[email protected]<mailto:[email protected]><mailto:
> [email protected]<mailto:[email protected]>>" <[email protected]
> <mailto:[email protected]><mailto:[email protected]<mailto:
> [email protected]>>>
> Subject: Re: How to configure/start multiple Input ports.
>
> Paresh,
> All Site-to-Site data comes across a single port connection. When
> adding a RPG to the source NiFi, the URL you provide will be the URL for
> the NCM of your internal NiFi cluster. You will add a a different RPG for
> each cluster you are connecting to. so in your case the external facing
> NiFi cluster will have 2 RPGs on the graph with 1 configured form the URL
> of the NCM on internal NiFi cluster 1 and the other for the NCM of internal
> NiFi cluster 2. Once a connection is established to an NCM, the NCM will
> communicate to the RPG the URLs and site-to-site ports for each of the
> clusters connected Nodes as well as the current load on those nodes. The
> NiFi with the RPG will then do a smart load-balanced delivery of data to
> those nodes. On the target cluster you will add input ports at the root
> level of the graph (They cannot be nested inside of any process groups) .
> You can add as many uniquely named input ports as you would like. These
> input ports will be exposed to the RPG on the sending system. When you
> draw a connection to the RPG, you will be provided witha pull-down
> selection menu of all available ports. Selecting one will complete this
> connection. So all data between NiFi systems via Site-to-Site will use the
> same site-to-site port but as you can see it enters the flow on the
> destination system via input ports.
>
> [Inline image 1]
>
> So on your externally visible NiFi you would have two RPGs setup like
> below:
> [Inline image 3]
>
> The receiving NiFi cluster will have one or more uniquely named input
> ports:
>
> [Inline image 4]
>
> As you can see from the above two screenshots the sending system's
> connection to the RPG shows that the connection as being connected to the
> input port "Node-Input" on the receiving cluster.
>
>
> Thanks,
> Matt
>
>
>
> On Wed, Jan 20, 2016 at 2:59 PM, Paresh Shah <[email protected]
> <mailto:[email protected]><mailto:[email protected]<mailto:
> [email protected]>>> wrote:
>
> We are trying to see how to run our pipelines in a clustered env using
> site-to-site. Our scenario is something like the following.
>
> 1. 3 different clusters.
> 2. One of the clusters is externally visible and will be the primary
> cluster.
> 3. All the data receivers will run on the primary node.
>
> Each of the pipelines has a RemoteProcessorGroup( RPG ) that would in
> –turn forward the data to their corresponding Input port.
>
> Looking at the way the site-to-site configuration indicates that we are
> only able to specify one port that would be started on a particular node.
> So the question is how can we pin a particular RPG to it corresponding
> InputPort.
>
> Any insights would be greatly appreciated.
>
> Thanks
> Paresh
> ________________________________
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ________________________________
>
> ________________________________
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ________________________________
>
> ________________________________
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ________________________________
>
