Jim The case you outline is quite common, critical for a system like this to handle well, and is something nifi offers some good options for.
So for a processor like PutFile it is quite fine to loop failure back to itself. What you want to do is consider if you want guaranteed delivery and if so how much space on nifi do you have to buffer up content. If you have say 100 GB of storage space you might want to set back pressure on that failure relationship to 50GB meaning you're willing to let it hold up to 50GB of data. Then things which would feed PutFile more data would stop sending more data until the condition improves. You could do things like send alerts based on number of failures in a period of time. You could use external monitoring tools or scripts to watch queue sizes and alert. You have a lot of options I guess is the key point. This is a great place to ask such questions. Welcome to the community. Please do have a read through the user guide if you've not already as that might also help to understand some of these considerations. Thanks Joe On Wed, Aug 24, 2016 at 9:17 AM, James McMahon <[email protected]> wrote: > > Hello. Relatively new NiFi developer here. I have a question borne from my > lack of experience. Can anyone offer any guidance? > > If a flowfile fails in a processor - let's say a Put File fails because of > underlying disk device failure - how do I replay the original flowfile(s)? I > need to understand how I resubmit the many files that would fail if, over > the course of a weekend, the disk device fails. Logging an attribute for an > error condition is a good start, but more important is resubmitting the body > of flowfiles when I get in Monday morning and reconcile the degraded disk > condition. > > Generally speaking, is it considered a good practice or a bad practice to > feed back into a processor all its failure condition flowfiles? I'm > concerned that this would lead to a bottleneck or race condition that could > quickly snowball if failed flowfiles are just fed back into the processor > continuously. What can I do to mitigate such a circumstance? What is > considered NiFi "best practice" to handle such failures? > > > > Apologies if this is not the proper venue to ask, and appreciate any > redirection advice. Am just starting to learn the ropes. > > Thanks very much. > >
