Oh - well - wow. That makes a lot of sense! Thank you Mark, I will try
that.
-Joe
On 6/18/2021 1:54 PM, Mark Payne wrote:
Joe,
The Retry processor should only be used if your intent is to do
something like “try 5 times and then do something else.” If the intent
is to just keep trying and avoid data loss, you can just route the
failure relationship from InvokeHTTP back to InvokeHTTP with nothing
in between. This is the typical pattern used in nifi.
Thanks
-Mark
Sent from my iPhone
On Jun 18, 2021, at 1:37 PM, Joe Obernberger
<[email protected]> wrote:
Thank you Arpad. We are using NiFi to call many REST endpoints with
lots of records. If one of the end points fails, we may not discover
it for a while, in which case these queues fill up. Then when the
endpoint comes back up, it won't process anything until we adjust the
queue size in NiFi.
Would love a better solution. I don't want to lose data, so I don't
want to drop the flowfiles. Basically, I just want it to stop
processing while the service (endpoint) is down, and then start up
again when the service comes back. I'm not familar with minifi; will
take a look.
On 6/18/2021 11:57 AM, Arpad Boda wrote:
Joe,
I think that's currently a limitation in backpressure handling of NiFi.
Single loopback connections are handled properly (those are
excluded), although more complex loopbacks (such as this retry
scenario) are not.
Although my gut feeling says that your configuration leaves space
for improvement. In case your both connections can fill up, you:
-May have too many retries
-Too big penalties for failures
-Simply the capacity of the connections are not big enough
What I would like to highlight here is that based on your average
data throughput and the maximum time you would like to keep retrying
sending the same flowfile, you should have a good estimation for the
number of flowfiles you can have in your loopback connections in
such a scenario. Apply some buffers and you should be good.
You can also apply maximum age for flowfiles, so they get dropped
after a while from these connections.
The long term solution would be to introduce a more complex logic to
handle loops when it comes to backpressure (such as the one in
minifi cpp), but I guess that will take a while. I'm pretty sure
that with a good flow design you can avoid this issue without it as
well.
Regards,
Arpad
On Thu, Jun 17, 2021 at 6:59 PM Joe Obernberger
<[email protected]> wrote:
Hi All - I'm wondering if there is an approach to using the
RetryFlowFile that doesn't get 'stuck' if there are lots of
failures.
I'm using InvokeHTTP and if it fails, the failure goes to a
RetryFlowFile process. The retry from the RetryFlowFile goes
back to
InvokeHTTP. If the InvokeHTTP process fails for a long time, the
failure queue fills, the retry queue fills, and when InvokeHTTP is
brought back up, it won't start since both queues are full.
Any ideas?
Thank you!
-Joe Obernberger
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
Virus-free. www.avg.com
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
