Re: Track dataflow with counter / updatecounter / statefull

[e-sociaux]

Hello Lehel,

 

Yes, I have already try to use UpdateAttribute with "Store State Locally".

The issue I saw it is when the same dataflow run multiple time the value "token" is increase all the times

I need to have the "counter" reset to Zero when the dataflow is consider to done.

 

Also, I have tested the Notify and Wait, but I remember it is not working in the cluster environment.

 

Anyway, I'm trying to deal with these informations.

 

Thanks

 

Minh

 

 

Envoyé: mercredi 21 février 2024 à 16:49
De: "Lehel Boér" <lehe...@hotmail.com>
À: "users@nifi.apache.org" <users@nifi.apache.org>
Objet: Re: Track dataflow with counter / updatecounter / statefull

Hi,

 

In NiFi, counters serve monitoring purposes but are not directly accessible through _expression_ language, and their values are lost upon restart. Instead, I suggest using an UpdateAttribute processor for this task. Configure it to 'Store State Locally' with an initial value of 0. Add a dynamic attribute, let's name it 'counter', using the _expression_ ${getStateValue("token"):plus(1)}. This _expression_ appends the incremented value to every flowfile, increasing it by 1 each time. Next, utilize the RouteOnAttributeProcessor to route flowfiles based on conditions such as '${token:equals(n)}'. Finally, use the ReplaceText processor to replace the flowfile content with the desired text intended for transmission to your Pub/Sub.

 

You can find more about trigger-based data processing in NiFi here.

 

Best Regards,

Lehel

 

---

From: e-soci...@gmx.fr <e-soci...@gmx.fr>
Sent: Wednesday, February 21, 2024 7:59
To: usersnifi.apache.org <users@nifi.apache.org>
Subject: Track dataflow with counter / updatecounter / statefull

 

 

Hello all,

 

Somebody has already use the Counter or UpdateCounter ?

 

My use case is sending a message (success) to topic pubsub if my nifi dataflow has consider to finish.

 

The input of dataflow is for example 3 flowfiles (do so treatements) and the ouput will be push to the GCS Bucket.

I want to have a message in the topic pubsub when I'm sure to have thoses 3 files in the bucket.

 

I'm trying use the updateCounter or statefull with updateAttribute processor but the "counter" is the owner of the processor so it is a bit difficult to update this counter.

 

Have you any information for me about this thing ?

 

Thanks all

 

Minh