Andrew, I will look at using the ControlRate processor as that might be a better alternative to throttling my workflow because I really need the outflow to be predictable.
Also, thanks Joe, I will be happy to test the change to UpdateAttribute – I have a test VM ready to go. Thanks everyone for your input. Regards, Kevin From: Andrew Grande [mailto:[email protected]] Sent: Thursday, April 27, 2017 5:08 PM To: [email protected] Subject: Re: Back Pressure Object threshold not honored I would also suggest the ControlRate processor in front of a sensitive step. There's a chance data may accumulate and sent in burst after downtime, so it ensures a predictable outflow. Andrew On Thu, Apr 27, 2017, 8:03 PM Joe Percivall <[email protected]<mailto:[email protected]>> wrote: Hello Kevin, I believe there are two things at play here. The first is the processor being very fast and processing the FlowFiles before back pressure gets applied. The second is that in the current distribution, UpdateAttribute uses an old style of getting higher performance and grabs batches of 100 with each onTrigger[1]. Since back-pressure gets applied per onTrigger the UpdateAttribute will process at least 100 FlowFiles before it gets told to stop processing. In the changes for 1.2.0 though I updated it to bring only 1 FlowFile in per onTrigger. So if you test this on a build of master then you should see more appropriate back-pressure application. [1] https://github.com/apache/nifi/blob/rel/nifi-1.1.2/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338 Joe On Thu, Apr 27, 2017 at 7:21 PM, Kevin Verhoeven <[email protected]<mailto:[email protected]>> wrote: Thank you for your help Andy. I think you are correct, the flowfiles are very small and the previous Processor is very fast – this might explain what is happening. I’ve enclosed screenshots of the connection properties and the workflow. In the screenshot I see 400 flowfiles were allowed through before back pressure was applied. The back pressure object threshold is set to 1. Do you have any recommendations? Kevin [cid:[email protected]] [cid:[email protected]] From: Andy LoPresto [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, April 27, 2017 4:16 PM To: [email protected]<mailto:[email protected]> Subject: Re: Back Pressure Object threshold not honored Hi Kevin, Sorry to hear you are having this issue. Can you please provide a screenshot of the connection properties in the configuration dialog? How quickly do those flowfiles get enqueued? I think there’s a chance if they are very small & the previous processor is very fast (i.e. RouteOnAttribute, SplitText) that it could enqueue a higher number before the back pressure check is executed. Andy LoPresto [email protected]<mailto:[email protected]> [email protected]<mailto:[email protected]> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven <[email protected]<mailto:[email protected]>> wrote: I have an odd problem. I set the Back Pressure Object threshold on a link between two Processors to 1, but 200 flowfiles are passed to the queue before back pressure is honored. I need the back pressure to be set to a small number of flowfiles to keep the source from flooding the destination. Has anyone come across this problem before? I am running 12 instances of NiFi on version 1.1.1 on Ubuntu 14.04. Regards, Kevin -- Joe Percivall e: [email protected]<mailto:[email protected]>
