Re: Wait/Notify inconsistent behavior
Hi Louis,
Glad to hear that works.
There are two approaches I can think of, to accumulate and combine
multiple service results.
The goal is creating a single FlowFile containing all service call
results, right?
1. Use DistributedCache(DC) as a shared heap space
- Each part of flow calls external services and store its result into DC
- Use cache keys in something like "OriginalRequestID_serviceType"
format. For example, "OriginalIncomingFlowFileUUID_serviceA"
- Use notify as well to let Wait know progress. Use serviceType as counter name
- After Wait finishes, use LookupAttribute or LookupRecord to enrich
the original FlowFile by fetching the accumulated results from DC
using DistributedMapCacheLookupService
2. Use MergeContent processor
- Route the original FlowFile into multiple sub-flows, meaning croning
the same FlowFile
- At each sub flow, call external service, update result FlowFile's
attribute so that MergeContent can wait and merge them:
-- fragment.identifier: some common id
-- fragment.count: the total number of sub-flows
-- fragment.index: to control merge order
- Route each resulting FlowFile into the same MergeContent processor,
use 'Defragment' Merge Strategy to merge FlowFiles based on
'fragment.*' attributes
-- Use Header, Footer and Demarcator intelligently to combine
contents, e.g. to form a valid JSON... etc
I'm not sure how do you want to loop, but here is another example flow
doing a traditional 'for' loop in NiFi.
https://gist.github.com/ijokarumawak/01c4fd2d9291d3e74ec424a581659ca8
Hope these can be helpful.
Thanks,
Koji
On Wed, Jan 9, 2019 at 1:04 AM Luis Carmona wrote:
>
>
> Hi Koji,
>
> tryed with some manual tests and seems to be working now, doesn't have the
> problema I had before. Today I will try it with a massvice flow and that will
> be the final check.
>
> Thanks for your help. Do you know where I can get a sample about processing
> in a loop, I mean, send things to a server, wait the answer on that, send to
> a second server accumulating the answers of both, and all this in a finite
> loop determined by the answers, Gathering all the answers in one final Json.
>
>
> Thank you very much.
>
> LC
>
>
>
> - Mensaje original -
> De: "Koji Kawamura"
> Para: "users"
> Enviados: Lunes, 7 de Enero 2019 22:22:12
> Asunto: Re: Wait/Notify inconsistent behavior
>
> Hi Luis,
>
> Look forward to know how it goes with a consolidated single Notify instance.
> Another benefit of doing so is by increasing 'Signal Buffer Count',
> the number of updates against the Distributed Cache Service can be
> decreased in your case, because multiple FlowFiles share the same
> signal id. Notify processor can merge counter deltas locally then
> update the cache entry only once.
>
> Thanks,
> Koji
>
> On Tue, Jan 8, 2019 at 3:50 AM Luis Carmona wrote:
> >
> > Hi Koji,
> >
> > thanks for taking the time to answer my question
> >
> > In the Wait processor:
> > - Signal CounterName is empty (default).
> > - Target Signal Count is set to 2
> >
> > About the Notify processor, I used two of them thinking about that
> > previously I set differently ${filename} in the precedings
> > UpdateAttribute(s).
> >
> > I attached the image of both processors, and the template as well.
> >
> >
> > The whole point of my layout is to send things to a server, wait the answer
> > on that, send to a second server accumulating the answers of both, and all
> > this in a finite loop determined by the answers, Gathering all the answers
> > in one final Json.
> >
> > By now Im stuck in the wait/notify issue, thanks for the sample I'll look
> > into it. Then I will see how to get the loop.
> >
> > Thanks a lot,
> >
> > Regards,
> >
> > LC
> >
> >
> > - Mensaje original -
> > De: "Koji Kawamura"
> > Para: "users"
> > Enviados: Domingo, 6 de Enero 2019 23:42:56
> > Asunto: Re: Wait/Notify inconsistent behavior
> >
> > The reason tu put two Notify processors is that I'm using different
> >
> > Hi Luis,
> >
> > Just a quick question, how are the "Signal Counter Name" and "Target
> > Signal Count" properties for the Wait processor configured?
> > If you'd like to wait the two sub-flow branches to complete, then:
> > "Signal Counter Name" should be blank, meaning check total count for
> > all counter names
> > "Target Signal Count" should be 2.
> >
> > If those are configured like that, then would you be able to share
> > your flow as a template for further investigation?
> >
> > One more thing, although Notify processor cares about atomicity, due
> > to the underlying distributed cache mechanism, concurrent writes to
> > the same cache identifier can overwrite existing signal, meaning one
> > of the two notifications can be lost.
> > To avoid this, using the same Notify instance at 3a and 3b in your
> > flow is highly recommended.
> > Here is an example flow to do that:
> > https://gist.github.com/ijokarumawak/6da4bd914236e941071cad103e1186dd
> >
> > Thanks,
> > Koji
> >
> > On Sat, Jan 5, 2019
Re: process group name reverts back to initial value if I do a nifi registry "Change version"
Bryan/Josef, Just wanted to let you know that I just tested this with NiFi 1.8.0 and NiFi Registry 0.3.0 and I experience the same behavior as Josef. -Chad > On Jan 8, 2019, at 12:58 PM, Bryan Bende wrote: > > I will keep trying, but I haven't been able to reproduce using NiFi > 1.8.0 and registry 0.2.0. > > I must be missing something. > > On Tue, Jan 8, 2019 at 11:50 AM wrote: >> >> I've tried it now on another secured cluster exactly like you did: >> >>1) Create PG "A" and save to registry >>2) Import PG "A" from registry and rename to "B" >>3) Add new processor (Execute Script) to original PG "A" and save to >> registry >>4) Change version on PG "B" to new version >> >> Problem still there... after changing to new version on "B" the name changed >> back to "A" >> >> >> >> On 08.01.19, 17:40, "Zahner Josef, GSB-LR-TRW-LI" >> wrote: >> >>Hi Bryan >> >>In my case it happens all the time, doesn't matter what kind of change. >> On my first test below I've changed a variable on a processor inside the PG >> and the second time (a few seconds ago) I've added a connection to my >> "Execute Script" processor. All the time my second PG with the new name >> changed back to the initial name... Even if I just click "Change version" >> and select another one than the current, my second PG changes the name back >> to the initial value. >> >>Btw. we use NiFi Registry 0.2.0. >> >>Cheers Josef >> >> >> >> >> >>On 08.01.19, 17:23, "Bryan Bende" wrote: >> >>Hi Josef, >> >>That sounds like a possible bug. I think the PG name is supposed to >>remain unchanged. >> >>I wasn't able to reproduce this though... in step 5 when you change >>the "abc" group, what type of change are you making? >> >>I did the following... >> >>1) Create PG "A" and save to registry >>2) Import PG "A" from registry and rename to "B" >>3) Add new processor to original PG "A" and save to registry >>4) Change version on PG "B" to new version >> >>PG "B" is still named "B" at this point. >> >>On Tue, Jan 8, 2019 at 10:26 AM wrote: >>> >>> Hi guys >>> >>> >>> >>> I’ve faced again an (at least for me) unexpected behavior of NiFi 1.8.0 >>> together with NiFi Registry. >>> >>> >>> >>> Following use case: >>> >>> >>> >>> Create a process group with name “abc” and add some processors to the pg >>> Commit the pg to the NiFi registry >>> Create a new pg and import the pg from step 1 from the registry >>> Change the name of the new pg to “def” instead of “abc” – so far so good, >>> no change from registry point of view >>> Change the original pg “abc” from step 1 and commit the change to the >>> registry >>> Now we have change to the newest version for the pg “def” from step 4, as >>> it isn’t anymore up to date – but now in my case as soon as I’m changing >>> the version, the pg name gets changed back to “abc”. This happens all the >>> time if I change the version on a pg which has another name than on the >>> commit >>> >>> >>> >>> Any comments on this? We use the NiFi registry as well as templating >>> infrastructure, means we have several times the same pg but with different >>> variables and names on the same NiFi canvas. But with the actual behavior >>> this is very inconvenient… we have to memorize the name before we do the >>> “Change version” and then after execution we have to set it again. >>> >>> >>> >>> Cheers Josef >> >> >> >>
Re: process group name reverts back to initial value if I do a nifi registry "Change version"
I will keep trying, but I haven't been able to reproduce using NiFi 1.8.0 and registry 0.2.0. I must be missing something. On Tue, Jan 8, 2019 at 11:50 AM wrote: > > I've tried it now on another secured cluster exactly like you did: > > 1) Create PG "A" and save to registry > 2) Import PG "A" from registry and rename to "B" > 3) Add new processor (Execute Script) to original PG "A" and save to > registry > 4) Change version on PG "B" to new version > > Problem still there... after changing to new version on "B" the name changed > back to "A" > > > > On 08.01.19, 17:40, "Zahner Josef, GSB-LR-TRW-LI" > wrote: > > Hi Bryan > > In my case it happens all the time, doesn't matter what kind of change. > On my first test below I've changed a variable on a processor inside the PG > and the second time (a few seconds ago) I've added a connection to my > "Execute Script" processor. All the time my second PG with the new name > changed back to the initial name... Even if I just click "Change version" and > select another one than the current, my second PG changes the name back to > the initial value. > > Btw. we use NiFi Registry 0.2.0. > > Cheers Josef > > > > > > On 08.01.19, 17:23, "Bryan Bende" wrote: > > Hi Josef, > > That sounds like a possible bug. I think the PG name is supposed to > remain unchanged. > > I wasn't able to reproduce this though... in step 5 when you change > the "abc" group, what type of change are you making? > > I did the following... > > 1) Create PG "A" and save to registry > 2) Import PG "A" from registry and rename to "B" > 3) Add new processor to original PG "A" and save to registry > 4) Change version on PG "B" to new version > > PG "B" is still named "B" at this point. > > On Tue, Jan 8, 2019 at 10:26 AM wrote: > > > > Hi guys > > > > > > > > I’ve faced again an (at least for me) unexpected behavior of NiFi > 1.8.0 together with NiFi Registry. > > > > > > > > Following use case: > > > > > > > > Create a process group with name “abc” and add some processors to > the pg > > Commit the pg to the NiFi registry > > Create a new pg and import the pg from step 1 from the registry > > Change the name of the new pg to “def” instead of “abc” – so far so > good, no change from registry point of view > > Change the original pg “abc” from step 1 and commit the change to > the registry > > Now we have change to the newest version for the pg “def” from step > 4, as it isn’t anymore up to date – but now in my case as soon as I’m > changing the version, the pg name gets changed back to “abc”. This happens > all the time if I change the version on a pg which has another name than on > the commit > > > > > > > > Any comments on this? We use the NiFi registry as well as > templating infrastructure, means we have several times the same pg but with > different variables and names on the same NiFi canvas. But with the actual > behavior this is very inconvenient… we have to memorize the name before we do > the “Change version” and then after execution we have to set it again. > > > > > > > > Cheers Josef > > > >
Re: process group name reverts back to initial value if I do a nifi registry "Change version"
I've tried it now on another secured cluster exactly like you did: 1) Create PG "A" and save to registry 2) Import PG "A" from registry and rename to "B" 3) Add new processor (Execute Script) to original PG "A" and save to registry 4) Change version on PG "B" to new version Problem still there... after changing to new version on "B" the name changed back to "A" On 08.01.19, 17:40, "Zahner Josef, GSB-LR-TRW-LI" wrote: Hi Bryan In my case it happens all the time, doesn't matter what kind of change. On my first test below I've changed a variable on a processor inside the PG and the second time (a few seconds ago) I've added a connection to my "Execute Script" processor. All the time my second PG with the new name changed back to the initial name... Even if I just click "Change version" and select another one than the current, my second PG changes the name back to the initial value. Btw. we use NiFi Registry 0.2.0. Cheers Josef On 08.01.19, 17:23, "Bryan Bende" wrote: Hi Josef, That sounds like a possible bug. I think the PG name is supposed to remain unchanged. I wasn't able to reproduce this though... in step 5 when you change the "abc" group, what type of change are you making? I did the following... 1) Create PG "A" and save to registry 2) Import PG "A" from registry and rename to "B" 3) Add new processor to original PG "A" and save to registry 4) Change version on PG "B" to new version PG "B" is still named "B" at this point. On Tue, Jan 8, 2019 at 10:26 AM wrote: > > Hi guys > > > > I’ve faced again an (at least for me) unexpected behavior of NiFi 1.8.0 together with NiFi Registry. > > > > Following use case: > > > > Create a process group with name “abc” and add some processors to the pg > Commit the pg to the NiFi registry > Create a new pg and import the pg from step 1 from the registry > Change the name of the new pg to “def” instead of “abc” – so far so good, no change from registry point of view > Change the original pg “abc” from step 1 and commit the change to the registry > Now we have change to the newest version for the pg “def” from step 4, as it isn’t anymore up to date – but now in my case as soon as I’m changing the version, the pg name gets changed back to “abc”. This happens all the time if I change the version on a pg which has another name than on the commit > > > > Any comments on this? We use the NiFi registry as well as templating infrastructure, means we have several times the same pg but with different variables and names on the same NiFi canvas. But with the actual behavior this is very inconvenient… we have to memorize the name before we do the “Change version” and then after execution we have to set it again. > > > > Cheers Josef smime.p7s Description: S/MIME Cryptographic Signature
Re: process group name reverts back to initial value if I do a nifi registry "Change version"
Hi Bryan In my case it happens all the time, doesn't matter what kind of change. On my first test below I've changed a variable on a processor inside the PG and the second time (a few seconds ago) I've added a connection to my "Execute Script" processor. All the time my second PG with the new name changed back to the initial name... Even if I just click "Change version" and select another one than the current, my second PG changes the name back to the initial value. Btw. we use NiFi Registry 0.2.0. Cheers Josef On 08.01.19, 17:23, "Bryan Bende" wrote: Hi Josef, That sounds like a possible bug. I think the PG name is supposed to remain unchanged. I wasn't able to reproduce this though... in step 5 when you change the "abc" group, what type of change are you making? I did the following... 1) Create PG "A" and save to registry 2) Import PG "A" from registry and rename to "B" 3) Add new processor to original PG "A" and save to registry 4) Change version on PG "B" to new version PG "B" is still named "B" at this point. On Tue, Jan 8, 2019 at 10:26 AM wrote: > > Hi guys > > > > I’ve faced again an (at least for me) unexpected behavior of NiFi 1.8.0 together with NiFi Registry. > > > > Following use case: > > > > Create a process group with name “abc” and add some processors to the pg > Commit the pg to the NiFi registry > Create a new pg and import the pg from step 1 from the registry > Change the name of the new pg to “def” instead of “abc” – so far so good, no change from registry point of view > Change the original pg “abc” from step 1 and commit the change to the registry > Now we have change to the newest version for the pg “def” from step 4, as it isn’t anymore up to date – but now in my case as soon as I’m changing the version, the pg name gets changed back to “abc”. This happens all the time if I change the version on a pg which has another name than on the commit > > > > Any comments on this? We use the NiFi registry as well as templating infrastructure, means we have several times the same pg but with different variables and names on the same NiFi canvas. But with the actual behavior this is very inconvenient… we have to memorize the name before we do the “Change version” and then after execution we have to set it again. > > > > Cheers Josef smime.p7s Description: S/MIME Cryptographic Signature
Re: process group name reverts back to initial value if I do a nifi registry "Change version"
Hi Josef, That sounds like a possible bug. I think the PG name is supposed to remain unchanged. I wasn't able to reproduce this though... in step 5 when you change the "abc" group, what type of change are you making? I did the following... 1) Create PG "A" and save to registry 2) Import PG "A" from registry and rename to "B" 3) Add new processor to original PG "A" and save to registry 4) Change version on PG "B" to new version PG "B" is still named "B" at this point. On Tue, Jan 8, 2019 at 10:26 AM wrote: > > Hi guys > > > > I’ve faced again an (at least for me) unexpected behavior of NiFi 1.8.0 > together with NiFi Registry. > > > > Following use case: > > > > Create a process group with name “abc” and add some processors to the pg > Commit the pg to the NiFi registry > Create a new pg and import the pg from step 1 from the registry > Change the name of the new pg to “def” instead of “abc” – so far so good, no > change from registry point of view > Change the original pg “abc” from step 1 and commit the change to the registry > Now we have change to the newest version for the pg “def” from step 4, as it > isn’t anymore up to date – but now in my case as soon as I’m changing the > version, the pg name gets changed back to “abc”. This happens all the time if > I change the version on a pg which has another name than on the commit > > > > Any comments on this? We use the NiFi registry as well as templating > infrastructure, means we have several times the same pg but with different > variables and names on the same NiFi canvas. But with the actual behavior > this is very inconvenient… we have to memorize the name before we do the > “Change version” and then after execution we have to set it again. > > > > Cheers Josef
process group name reverts back to initial value if I do a nifi registry "Change version"
Hi guys I’ve faced again an (at least for me) unexpected behavior of NiFi 1.8.0 together with NiFi Registry. Following use case: 1. Create a process group with name “abc” and add some processors to the pg 2. Commit the pg to the NiFi registry 3. Create a new pg and import the pg from step 1 from the registry 4. Change the name of the new pg to “def” instead of “abc” – so far so good, no change from registry point of view 5. Change the original pg “abc” from step 1 and commit the change to the registry 6. Now we have change to the newest version for the pg “def” from step 4, as it isn’t anymore up to date – but now in my case as soon as I’m changing the version, the pg name gets changed back to “abc”. This happens all the time if I change the version on a pg which has another name than on the commit Any comments on this? We use the NiFi registry as well as templating infrastructure, means we have several times the same pg but with different variables and names on the same NiFi canvas. But with the actual behavior this is very inconvenient… we have to memorize the name before we do the “Change version” and then after execution we have to set it again. Cheers Josef smime.p7s Description: S/MIME Cryptographic Signature
