Hello fellow NiFi users, we have some fairly large NiFi flows and would like to monitor whether there are FlowFiles stucked, e.g. because of failing processors or being stuck in a loop.
Is there a way to find any FlowFiles being "stuck" in the Flow for longer than a time period of X, e.g. an hour at any part of the flow? I'm aware of the concept of "Lineage duration" but could not find a REST endpoint or something similar that would allow this to be checked. We're not interested in monitoring overall processing times but rather want to ensure that all FlowFiles are processed successfully within a certain time frame. *Is there a way to get the lineage duration of all FlowFiles currently in the whole flow?* We thought about crawling the REST api for every connection and listing their queues but this seems rather inconvenient. Also we would not be able to track down files currently being processed by a processor. Is there an easier way? Monitoring the provenance events does not seem to be an option as this would miss FlowFiles being stuck in a queue for a long period of time as this does not create a provenance event. Also we would not be able to a) pin-point where the FlowFile is at and b) more importantly determine whether the FlowFile is still in the flow or processed successfully yet. Is there some alternative? Has anyone an idea whether this would be hard to bring to NiFi itself? I'm not familiar enough with the source code of NiFi to assess the difficulty. Any hints are highly appreciated! Thank you. :)
