Re: Data migration in sequence

[Mike Thomsen]

> Hi Joe, The systems are the same. A and B. Just that i will be pulling 
> different
data set and that has to be done in sequence

If your input is JSON that is being sent to the originating API to get
different data sets, what you can do is put all of the GenerateFlowFile
instances' content into one big JSON array and have SplitJson split the
array into multiple flowfiles using the JSONPath statement "$" without the
quotes.

On Thu, Apr 12, 2018 at 12:56 PM Anil Rai  wrote:

> Thanks Mike and Joe for the quick response.
> Hi Mike, Nifi is making api calls to source -> pull data -> Maps -> Post
> data to target via API. Generate flow file triggers the flow. Right now i
> manually run 1 process group after the previous one is complete.
> Hi Joe, The systems are the same. A and B. Just that i will be pulling
> different data set and that has to be done in sequence. Example, I will
> load customer followed by say order followed by items etc.
> The use case is one time data load from system A and B of different data
> sets that needs to happen in sequence. Trigger has to be only once to start
> the first one.
> Hope that helps.
>
> Regards
> Anil
>
>
> On Thu, Apr 12, 2018 at 12:26 PM, Joe Witt  wrote:
>
> > Anil
> >
> > What is a little hard to tell here is do you mean:
> >
> > 1) I have to transfer data from system A to system B where there are
> > 10 steps I want to execute in NiFi between A and B.
> > -or-
> > 2) I have to transfer data from systems A,B,C,D,E,F,G,H,I,J to system
> > Z where first I want to load data from system A to Z, then B to Z,
> > then C to Z and so on.
> >
> > If it is #1 then of course it is easy.
> > If it is #2 then I believe you'd want a series of Wait/Notify
> > processors where B waits the notification signal from A, C from B, and
> > so on.
> >
> > Thanks
> >
> > On Thu, Apr 12, 2018 at 4:16 PM, Mike Thomsen 
> > wrote:
> > > It'd be very helpful to know what the input and output processors are
> for
> > > each of the flows because they have different capabilities for
> triggering
> > > them.
> > >
> > > On Thu, Apr 12, 2018 at 12:03 PM Anil Rai 
> wrote:
> > >
> > >> sorry, sent email in between.
> > >> This can be achieved by putting few flags in cache and checking them
> in
> > >> subsequent flows. But this is not elegant.
> > >> Any other way to achieve this flow runs in a sequence?
> > >>
> > >> Thanks
> > >> Anil
> > >>
> > >>
> > >> On Thu, Apr 12, 2018 at 12:01 PM, Anil Rai 
> > wrote:
> > >>
> > >> > Experts, Below is my requirenment.
> > >> >
> > >> > I have to do a 1 time data load from system a to b.
> > >> > I have to transfer 10 different data sets.
> > >> > These transfers have to happen in sequence. 1 followed by 2 followed
> > by 3
> > >> > so on till 10.
> > >> >
> > >> > We have created seperate process groups for each of the data set
> > >> migration.
> > >> > How can we achieve the run sequence once I trigger the first flow?
> > >> > I know this can be achieved by putting few flags in cache and
> checking
> > >> > them in subsequen
> > >> >
> > >>
> >
>

Re: Data migration in sequence

[Anil Rai]

Thanks Mike and Joe for the quick response.
Hi Mike, Nifi is making api calls to source -> pull data -> Maps -> Post
data to target via API. Generate flow file triggers the flow. Right now i
manually run 1 process group after the previous one is complete.
Hi Joe, The systems are the same. A and B. Just that i will be pulling
different data set and that has to be done in sequence. Example, I will
load customer followed by say order followed by items etc.
The use case is one time data load from system A and B of different data
sets that needs to happen in sequence. Trigger has to be only once to start
the first one.
Hope that helps.

Regards
Anil


On Thu, Apr 12, 2018 at 12:26 PM, Joe Witt  wrote:

> Anil
>
> What is a little hard to tell here is do you mean:
>
> 1) I have to transfer data from system A to system B where there are
> 10 steps I want to execute in NiFi between A and B.
> -or-
> 2) I have to transfer data from systems A,B,C,D,E,F,G,H,I,J to system
> Z where first I want to load data from system A to Z, then B to Z,
> then C to Z and so on.
>
> If it is #1 then of course it is easy.
> If it is #2 then I believe you'd want a series of Wait/Notify
> processors where B waits the notification signal from A, C from B, and
> so on.
>
> Thanks
>
> On Thu, Apr 12, 2018 at 4:16 PM, Mike Thomsen 
> wrote:
> > It'd be very helpful to know what the input and output processors are for
> > each of the flows because they have different capabilities for triggering
> > them.
> >
> > On Thu, Apr 12, 2018 at 12:03 PM Anil Rai  wrote:
> >
> >> sorry, sent email in between.
> >> This can be achieved by putting few flags in cache and checking them in
> >> subsequent flows. But this is not elegant.
> >> Any other way to achieve this flow runs in a sequence?
> >>
> >> Thanks
> >> Anil
> >>
> >>
> >> On Thu, Apr 12, 2018 at 12:01 PM, Anil Rai 
> wrote:
> >>
> >> > Experts, Below is my requirenment.
> >> >
> >> > I have to do a 1 time data load from system a to b.
> >> > I have to transfer 10 different data sets.
> >> > These transfers have to happen in sequence. 1 followed by 2 followed
> by 3
> >> > so on till 10.
> >> >
> >> > We have created seperate process groups for each of the data set
> >> migration.
> >> > How can we achieve the run sequence once I trigger the first flow?
> >> > I know this can be achieved by putting few flags in cache and checking
> >> > them in subsequen
> >> >
> >>
>

Re: Data migration in sequence

[Joe Witt]

Anil

What is a little hard to tell here is do you mean:

1) I have to transfer data from system A to system B where there are
10 steps I want to execute in NiFi between A and B.
-or-
2) I have to transfer data from systems A,B,C,D,E,F,G,H,I,J to system
Z where first I want to load data from system A to Z, then B to Z,
then C to Z and so on.

If it is #1 then of course it is easy.
If it is #2 then I believe you'd want a series of Wait/Notify
processors where B waits the notification signal from A, C from B, and
so on.

Thanks

On Thu, Apr 12, 2018 at 4:16 PM, Mike Thomsen  wrote:
> It'd be very helpful to know what the input and output processors are for
> each of the flows because they have different capabilities for triggering
> them.
>
> On Thu, Apr 12, 2018 at 12:03 PM Anil Rai  wrote:
>
>> sorry, sent email in between.
>> This can be achieved by putting few flags in cache and checking them in
>> subsequent flows. But this is not elegant.
>> Any other way to achieve this flow runs in a sequence?
>>
>> Thanks
>> Anil
>>
>>
>> On Thu, Apr 12, 2018 at 12:01 PM, Anil Rai  wrote:
>>
>> > Experts, Below is my requirenment.
>> >
>> > I have to do a 1 time data load from system a to b.
>> > I have to transfer 10 different data sets.
>> > These transfers have to happen in sequence. 1 followed by 2 followed by 3
>> > so on till 10.
>> >
>> > We have created seperate process groups for each of the data set
>> migration.
>> > How can we achieve the run sequence once I trigger the first flow?
>> > I know this can be achieved by putting few flags in cache and checking
>> > them in subsequen
>> >
>>

Re: Data migration in sequence

[Mike Thomsen]

It'd be very helpful to know what the input and output processors are for
each of the flows because they have different capabilities for triggering
them.

On Thu, Apr 12, 2018 at 12:03 PM Anil Rai  wrote:

> sorry, sent email in between.
> This can be achieved by putting few flags in cache and checking them in
> subsequent flows. But this is not elegant.
> Any other way to achieve this flow runs in a sequence?
>
> Thanks
> Anil
>
>
> On Thu, Apr 12, 2018 at 12:01 PM, Anil Rai  wrote:
>
> > Experts, Below is my requirenment.
> >
> > I have to do a 1 time data load from system a to b.
> > I have to transfer 10 different data sets.
> > These transfers have to happen in sequence. 1 followed by 2 followed by 3
> > so on till 10.
> >
> > We have created seperate process groups for each of the data set
> migration.
> > How can we achieve the run sequence once I trigger the first flow?
> > I know this can be achieved by putting few flags in cache and checking
> > them in subsequen
> >
>