To add to the back pressure management options, we wrote a custom "RouteOnBackPressure" processor [1] that would allow us to re-route the incoming flow after outgoing queues hit their backpressure thresholds. The processor itself doesn't handle anything with expiration (though output queues could), and requires our third-party NAR bundle [2], but it works well for handling surges.
[1] https://github.com/Asymmetrik/nifi-nar-bundles/blob/develop/nifi-asymmetrik-standard-bundle/nifi-asymmetrik-standard-processors/src/main/java/com/asymmetrik/nifi/processors/RouteOnBackPressure.java [2] https://github.com/Asymmetrik/nifi-nar-bundles On Wed, Jul 24, 2019 at 9:18 AM Peter Wicks (pwicks) <[email protected]> wrote: > Thanks for the input Mark, I can definitely see that being valuable. That > leads to some new ideas. > > We could have an, "Expiration Strategy" drop down. Options might be: > > - Lineage Expiration (default, keeps backwards compat.) > - Queue Expiration > - Back Pressure + Queue Expiration (which would follow the rules Mark > described). > > As for the order, I'm not confident I remember how it works right now. > > Thanks, > Peter > > > -----Original Message----- > From: Mark Bean <[email protected]> > Sent: Wednesday, July 24, 2019 7:08 AM > To: [email protected] > Subject: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times > > On a similar note, we recently had a case where it would be desirable for > the flowfile expiration to kick in only after the flowfile size/count back > pressure limits have been reached. In other words, once a back pressure > (size) limit is reached, it would be desirable to then remove flowfiles - > beginning with the oldest first - until the back pressure limit is no > longer in violation. > > Thanks, > Mark > > On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) <[email protected]> > wrote: > > > I was thinking it would be nice to expire FlowFile's based on their > > time in queue, in addition to the current option of their total > > lineage time (as in, have both options available). > > Any thoughts on pros/cons of having this available? > > > > Thanks, > > Peter > > >
