I forgot to mention the bigger problem with the way this prioritization works. If three queues lead to a single component, and all 3 are full due to back pressure, it is easily observable that once Queue A starts to drain, it will continue to drain as long as it has incoming flow files. This means that if there's a constant flow of incoming flow files to Queue A, the other two queues will never be drained (at least, that's my theory based on an hour of observation).
I'm going to create a JIRA ticket to track this. On Thu, Sep 29, 2016 at 4:02 PM, Joe Gresock <[email protected]> wrote: > In my flow, I receive flow files from multiple sources that eventually get > funneled into one processor. The connections use OldestFirstPrioritizer, > and while there is no backpressure, the cross-queue prioritization appears > to work as designed (i.e., each queue that leads into the funnel appears to > be fairly drained, resulting in essentially a round robin distribution > among the funneled queues). > > However, as soon as backpressure engages, NiFi appears to want to fully > drain one of the funneled queues, leaving the others untouched. This > happens even when the undrained queues have flow files older than the ones > that are drained first. Is this known behavior in NiFi? If so, do you > guys think it should be addressed? > > -- > I know what it is to be in need, and I know what it is to have plenty. I > have learned the secret of being content in any and every situation, > whether well fed or hungry, whether living in plenty or in want. I can > do all this through him who gives me strength. *-Philippians 4:12-13* > -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength. *-Philippians 4:12-13*
