Jim,
You can actually reference “lastQueueDate” in Expression Language. It is
formatted as number of milliseconds since epoch.
So you might have a RouteOnAttribute that has a property named “old” with a
value of:
${lastQueueDate:lt( ${now():minus(10000)} )}
So any FlowFile that has been queued for more than 10 seconds would be routed
to “old”, anything else to “unmatched”
Thanks
-Mark
On Feb 15, 2024, at 10:18 AM, James McMahon <[email protected]> wrote:
That would work - what a good suggestion. I'll do that. I can format the
resulting number and then RouteOnAttribute by the desired subset of the result.
Something like this to set attribute dt.failure:
${now():toNumber():toDate("yyyy-MM-ddHH:mm:ss"):format("yyyyMMddHHmmss","EST")}
Then I can effectively route the files.
Thank you Jim S.
On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey
<[email protected]<mailto:[email protected]>> wrote:
You could add an UpdateAttribute processor first in the failure path to add a
new attribute which contains the time the error occurred by using the ${now()}
or ${now():toNumber()} expression language function.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now
Then later on in the flow you can compare current time to the saved error time
to see how much time has elapsed.
— Jim
On Feb 15, 2024, at 9:44 AM, James McMahon
<[email protected]<mailto:[email protected]>> wrote:
As it turns out lineageStartDate and Queued Duration are very different.
Without being able to get at Queued Duration as an attribute, it appears we
cannot RouteOnAttribute to filter thousands in a queue by anything like hours
they have been in queue.
Why would this be helpful? Let us say we have an InvokeHttp processor making
calls to a REST endpoint. We leave for a weekend and return to find 5000 files
in the Failure queue from this processor. It would be most helpful to identify
the start time and end time of these 5000 failures. We can't do that reviewing
only the first 100 flowfiles in the queue from the UI.
One can make an assumption that all of these 5000 flowfiles that failed
InvokeHttp share a similar range of lineageStartDate, but that will not
necessarily be true depending on flow complexity.
On Wed, Feb 14, 2024 at 9:49 AM James McMahon
<[email protected]<mailto:[email protected]>> wrote:
What a great workaround, thank you once again Mike. I'll put this in and use it
now.
Jim
On Tue, Feb 13, 2024 at 4:41 PM Michael Moser
<[email protected]<mailto:[email protected]>> wrote:
Hello James,
I'm not aware of a way to access Queued Duration using expression language, but
you can access the Lineage Duration information. The Getting Started Guide
mentions both entryDate and lineageStartDate as immutable attributes on all
flowfiles. These are numbers of milliseconds since epoch. If you need them in
a readable format, you can use the format() function.
simple examples:
${entryDate} = 1707859943778
${lineageStartDate} = 1707859943778
${lineageStartDate:format("yyyy-MM-dd HH:mm:ss.SSS")} = 2024-02-13 21:32:23.778
-- Mike
On Mon, Feb 12, 2024 at 11:38 AM James McMahon
<[email protected]<mailto:[email protected]>> wrote:
When we examine the contents of a queue through the UI and select a flowfile
from the resulting list, we see FlowFile Details in the Details tab. Are those
key/values accessible from nifi expression language? I would like to access
Queued Duration. I have a queue that holds flowfiles with non-successful return
codes for calls to REST services, and I want to route depending on how long
these flowfiles have been sitting in my error queue to isolate the window when
the REST service was unavailable.
Thank you for any examples that show how we can access these keys and values.
