Joe, Thanks for clarifying since I had seen the same thing. And chalked it up to research further if object meant same thing as FlowFile ....
I just mention it because it's first instance that I notice when object is implied as FlowFile and it is confusing. Thx On Thu, Apr 27, 2017 at 8:16 PM Kevin Verhoeven <[email protected]> wrote: > Andrew, I will look at using the ControlRate processor as that might be a > better alternative to throttling my workflow because I really need the > outflow to be predictable. > > > > Also, thanks Joe, I will be happy to test the change to UpdateAttribute – > I have a test VM ready to go. > > > > Thanks everyone for your input. > > > > Regards, > > > > Kevin > > > > *From:* Andrew Grande [mailto:[email protected]] > *Sent:* Thursday, April 27, 2017 5:08 PM > > > *To:* [email protected] > *Subject:* Re: Back Pressure Object threshold not honored > > > > I would also suggest the ControlRate processor in front of a sensitive > step. There's a chance data may accumulate and sent in burst after > downtime, so it ensures a predictable outflow. > > Andrew > > > > On Thu, Apr 27, 2017, 8:03 PM Joe Percivall <[email protected]> > wrote: > > Hello Kevin, > > > > I believe there are two things at play here. The first is the processor > being very fast and processing the FlowFiles before back pressure gets > applied. The second is that in the current distribution, UpdateAttribute > uses an old style of getting higher performance and grabs batches of 100 > with each onTrigger[1]. Since back-pressure gets applied per onTrigger the > UpdateAttribute will process at least 100 FlowFiles before it gets told to > stop processing. > > > > In the changes for 1.2.0 though I updated it to bring only 1 FlowFile in > per onTrigger. So if you test this on a build of master then you should see > more appropriate back-pressure application. > > > > [1] > https://github.com/apache/nifi/blob/rel/nifi-1.1.2/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338 > > > > Joe > > > > On Thu, Apr 27, 2017 at 7:21 PM, Kevin Verhoeven < > [email protected]> wrote: > > Thank you for your help Andy. I think you are correct, the flowfiles are > very small and the previous Processor is very fast – this might explain > what is happening. I’ve enclosed screenshots of the connection properties > and the workflow. In the screenshot I see 400 flowfiles were allowed > through before back pressure was applied. The back pressure object > threshold is set to 1. Do you have any recommendations? > > > > Kevin > > > > > > > > *From:* Andy LoPresto [mailto:[email protected]] > *Sent:* Thursday, April 27, 2017 4:16 PM > *To:* [email protected] > *Subject:* Re: Back Pressure Object threshold not honored > > > > Hi Kevin, > > > > Sorry to hear you are having this issue. Can you please provide a > screenshot of the connection properties in the configuration dialog? How > quickly do those flowfiles get enqueued? I think there’s a chance if they > are very small & the previous processor is very fast (i.e. > RouteOnAttribute, SplitText) that it could enqueue a higher number before > the back pressure check is executed. > > > > Andy LoPresto > > [email protected] > > *[email protected] <[email protected]>* > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven <[email protected]> > wrote: > > > > I have an odd problem. I set the Back Pressure Object threshold on a link > between two Processors to 1, but 200 flowfiles are passed to the queue > before back pressure is honored. I need the back pressure to be set to a > small number of flowfiles to keep the source from flooding the destination. > Has anyone come across this problem before? I am running 12 instances of > NiFi on version 1.1.1 on Ubuntu 14.04. > > > > Regards, > > > > Kevin > > > > > > > > -- > > *Joe Percivall* > > e: [email protected] > >
