Hello Kevin, I believe there are two things at play here. The first is the processor being very fast and processing the FlowFiles before back pressure gets applied. The second is that in the current distribution, UpdateAttribute uses an old style of getting higher performance and grabs batches of 100 with each onTrigger[1]. Since back-pressure gets applied per onTrigger the UpdateAttribute will process at least 100 FlowFiles before it gets told to stop processing.
In the changes for 1.2.0 though I updated it to bring only 1 FlowFile in per onTrigger. So if you test this on a build of master then you should see more appropriate back-pressure application. [1] https://github.com/apache/nifi/blob/rel/nifi-1.1.2/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338 Joe On Thu, Apr 27, 2017 at 7:21 PM, Kevin Verhoeven <[email protected]> wrote: > Thank you for your help Andy. I think you are correct, the flowfiles are > very small and the previous Processor is very fast – this might explain > what is happening. I’ve enclosed screenshots of the connection properties > and the workflow. In the screenshot I see 400 flowfiles were allowed > through before back pressure was applied. The back pressure object > threshold is set to 1. Do you have any recommendations? > > > > Kevin > > > > > > > > *From:* Andy LoPresto [mailto:[email protected]] > *Sent:* Thursday, April 27, 2017 4:16 PM > *To:* [email protected] > *Subject:* Re: Back Pressure Object threshold not honored > > > > Hi Kevin, > > > > Sorry to hear you are having this issue. Can you please provide a > screenshot of the connection properties in the configuration dialog? How > quickly do those flowfiles get enqueued? I think there’s a chance if they > are very small & the previous processor is very fast (i.e. > RouteOnAttribute, SplitText) that it could enqueue a higher number before > the back pressure check is executed. > > > > Andy LoPresto > > [email protected] > > *[email protected] <[email protected]>* > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven <[email protected]> > wrote: > > > > I have an odd problem. I set the Back Pressure Object threshold on a link > between two Processors to 1, but 200 flowfiles are passed to the queue > before back pressure is honored. I need the back pressure to be set to a > small number of flowfiles to keep the source from flooding the destination. > Has anyone come across this problem before? I am running 12 instances of > NiFi on version 1.1.1 on Ubuntu 14.04. > > > > Regards, > > > > Kevin > > > -- *Joe Percivall* e: [email protected]
