That would work - what a good suggestion. I'll do that. I can format the
resulting number and then RouteOnAttribute by the desired subset of the
result.
Something like this to set attribute dt.failure:
${now():toNumber():toDate("yyyy-MM-ddHH:mm:ss"):format("yyyyMMddHHmmss","EST")}
Then I can effectively route the files.
Thank you Jim S.On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey <[email protected]> wrote: > You could add an UpdateAttribute processor first in the failure path to > add a new attribute which contains the time the error occurred by using the > ${now()} or ${now():toNumber()} expression language function. > > https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now > > Then later on in the flow you can compare current time to the saved error > time to see how much time has elapsed. > > — Jim > > > On Feb 15, 2024, at 9:44 AM, James McMahon <[email protected]> wrote: > > As it turns out lineageStartDate and Queued Duration are very different. > Without being able to get at Queued Duration as an attribute, it appears we > cannot RouteOnAttribute to filter thousands in a queue by anything like > hours they have been in queue. > Why would this be helpful? Let us say we have an InvokeHttp processor > making calls to a REST endpoint. We leave for a weekend and return to find > 5000 files in the Failure queue from this processor. It would be most > helpful to identify the start time and end time of these 5000 failures. We > can't do that reviewing only the first 100 flowfiles in the queue from the > UI. > One can make an assumption that all of these 5000 flowfiles that failed > InvokeHttp share a similar range of lineageStartDate, but that will not > necessarily be true depending on flow complexity. > > On Wed, Feb 14, 2024 at 9:49 AM James McMahon <[email protected]> > wrote: > >> What a great workaround, thank you once again Mike. I'll put this in and >> use it now. >> Jim >> >> On Tue, Feb 13, 2024 at 4:41 PM Michael Moser <[email protected]> wrote: >> >>> Hello James, >>> >>> I'm not aware of a way to access Queued Duration using expression >>> language, but you can access the Lineage Duration information. The Getting >>> Started Guide mentions both entryDate and lineageStartDate as immutable >>> attributes on all flowfiles. These are numbers of milliseconds since >>> epoch. If you need them in a readable format, you can use the format() >>> function. >>> >>> simple examples: >>> ${entryDate} = 1707859943778 >>> ${lineageStartDate} = 1707859943778 >>> ${lineageStartDate:format("yyyy-MM-dd HH:mm:ss.SSS")} = 2024-02-13 >>> 21:32:23.778 >>> >>> -- Mike >>> >>> >>> On Mon, Feb 12, 2024 at 11:38 AM James McMahon <[email protected]> >>> wrote: >>> >>>> When we examine the contents of a queue through the UI and select a >>>> flowfile from the resulting list, we see FlowFile Details in the Details >>>> tab. Are those key/values accessible from nifi expression language? I would >>>> like to access Queued Duration. I have a queue that holds flowfiles with >>>> non-successful return codes for calls to REST services, and I want to route >>>> depending on how long these flowfiles have been sitting in my error queue >>>> to isolate the window when the REST service was unavailable. >>>> Thank you for any examples that show how we can access these keys and >>>> values. >>>> >>> >
