Thank you very much for your reply Joe. I look forward to learning more from all the great questions I see on the users and developers lists, and to finding ways to contribute too.
Cheers, Jim On Wed, Aug 24, 2016 at 9:32 AM, Joe Witt <[email protected]> wrote: > Jim > > The case you outline is quite common, critical for a system like this > to handle well, and is something nifi offers some good options for. > > So for a processor like PutFile it is quite fine to loop failure back > to itself. What you want to do is consider if you want guaranteed > delivery and if so how much space on nifi do you have to buffer up > content. If you have say 100 GB of storage space you might want to > set back pressure on that failure relationship to 50GB meaning you're > willing to let it hold up to 50GB of data. Then things which would > feed PutFile more data would stop sending more data until the > condition improves. You could do things like send alerts based on > number of failures in a period of time. You could use external > monitoring tools or scripts to watch queue sizes and alert. You have > a lot of options I guess is the key point. > > This is a great place to ask such questions. Welcome to the > community. Please do have a read through the user guide if you've not > already as that might also help to understand some of these > considerations. > > Thanks > Joe > > On Wed, Aug 24, 2016 at 9:17 AM, James McMahon <[email protected]> > wrote: > > > > Hello. Relatively new NiFi developer here. I have a question borne from > my > > lack of experience. Can anyone offer any guidance? > > > > If a flowfile fails in a processor - let's say a Put File fails because > of > > underlying disk device failure - how do I replay the original > flowfile(s)? I > > need to understand how I resubmit the many files that would fail if, over > > the course of a weekend, the disk device fails. Logging an attribute for > an > > error condition is a good start, but more important is resubmitting the > body > > of flowfiles when I get in Monday morning and reconcile the degraded disk > > condition. > > > > Generally speaking, is it considered a good practice or a bad practice to > > feed back into a processor all its failure condition flowfiles? I'm > > concerned that this would lead to a bottleneck or race condition that > could > > quickly snowball if failed flowfiles are just fed back into the processor > > continuously. What can I do to mitigate such a circumstance? What is > > considered NiFi "best practice" to handle such failures? > > > > > > > > Apologies if this is not the proper venue to ask, and appreciate any > > redirection advice. Am just starting to learn the ropes. > > > > Thanks very much. > > > > >
