Joe, The Retry processor should only be used if your intent is to do something like “try 5 times and then do something else.” If the intent is to just keep trying and avoid data loss, you can just route the failure relationship from InvokeHTTP back to InvokeHTTP with nothing in between. This is the typical pattern used in nifi.
Thanks -Mark Sent from my iPhone On Jun 18, 2021, at 1:37 PM, Joe Obernberger <[email protected]> wrote: Thank you Arpad. We are using NiFi to call many REST endpoints with lots of records. If one of the end points fails, we may not discover it for a while, in which case these queues fill up. Then when the endpoint comes back up, it won't process anything until we adjust the queue size in NiFi. Would love a better solution. I don't want to lose data, so I don't want to drop the flowfiles. Basically, I just want it to stop processing while the service (endpoint) is down, and then start up again when the service comes back. I'm not familar with minifi; will take a look. On 6/18/2021 11:57 AM, Arpad Boda wrote: Joe, I think that's currently a limitation in backpressure handling of NiFi. Single loopback connections are handled properly (those are excluded), although more complex loopbacks (such as this retry scenario) are not. Although my gut feeling says that your configuration leaves space for improvement. In case your both connections can fill up, you: -May have too many retries -Too big penalties for failures -Simply the capacity of the connections are not big enough What I would like to highlight here is that based on your average data throughput and the maximum time you would like to keep retrying sending the same flowfile, you should have a good estimation for the number of flowfiles you can have in your loopback connections in such a scenario. Apply some buffers and you should be good. You can also apply maximum age for flowfiles, so they get dropped after a while from these connections. The long term solution would be to introduce a more complex logic to handle loops when it comes to backpressure (such as the one in minifi cpp), but I guess that will take a while. I'm pretty sure that with a good flow design you can avoid this issue without it as well. Regards, Arpad On Thu, Jun 17, 2021 at 6:59 PM Joe Obernberger <[email protected]<mailto:[email protected]>> wrote: Hi All - I'm wondering if there is an approach to using the RetryFlowFile that doesn't get 'stuck' if there are lots of failures. I'm using InvokeHTTP and if it fails, the failure goes to a RetryFlowFile process. The retry from the RetryFlowFile goes back to InvokeHTTP. If the InvokeHTTP process fails for a long time, the failure queue fills, the retry queue fills, and when InvokeHTTP is brought back up, it won't start since both queues are full. Any ideas? Thank you! -Joe Obernberger [https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
