Re: Back Pressure Object threshold not honored

[Juan Sequeiros]

Joe,

Thanks for clarifying since I had seen the same thing. And chalked it up to
research further if object meant same thing as FlowFile 

I just mention it because it's first instance that I notice when object is
implied as FlowFile and it is confusing.

Thx
On Thu, Apr 27, 2017 at 8:16 PM Kevin Verhoeven <kevin.verhoe...@ds-iq.com>
wrote:

> Andrew, I will look at using the ControlRate processor as that might be a
> better alternative to throttling my workflow because I really need the
> outflow to be predictable.
>
>
>
> Also, thanks Joe, I will be happy to test the change to UpdateAttribute –
> I have a test VM ready to go.
>
>
>
> Thanks everyone for your input.
>
>
>
> Regards,
>
>
>
> Kevin
>
>
>
> *From:* Andrew Grande [mailto:apere...@gmail.com]
> *Sent:* Thursday, April 27, 2017 5:08 PM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: Back Pressure Object threshold not honored
>
>
>
> I would also suggest the ControlRate processor in front of a sensitive
> step. There's a chance data may accumulate and sent in burst after
> downtime, so it ensures a predictable outflow.
>
> Andrew
>
>
>
> On Thu, Apr 27, 2017, 8:03 PM Joe Percivall <joeperciv...@gmail.com>
> wrote:
>
> Hello Kevin,
>
>
>
> I believe there are two things at play here. The first is the processor
> being very fast and processing the FlowFiles before back pressure gets
> applied. The second is that in the current distribution, UpdateAttribute
> uses an old style of getting higher performance and grabs batches of 100
> with each onTrigger[1]. Since back-pressure gets applied per onTrigger the
> UpdateAttribute will process at least 100 FlowFiles before it gets told to
> stop processing.
>
>
>
> In the changes for 1.2.0 though I updated it to bring only 1 FlowFile in
> per onTrigger. So if you test this on a build of master then you should see
> more appropriate back-pressure application.
>
>
>
> [1]
> https://github.com/apache/nifi/blob/rel/nifi-1.1.2/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338
>
>
>
> Joe
>
>
>
> On Thu, Apr 27, 2017 at 7:21 PM, Kevin Verhoeven <
> kevin.verhoe...@ds-iq.com> wrote:
>
> Thank you for your help Andy. I think you are correct, the flowfiles are
> very small and the previous Processor is very fast – this might explain
> what is happening. I’ve enclosed screenshots of the connection properties
> and the workflow. In the screenshot I see 400 flowfiles were allowed
> through before back pressure was applied. The back pressure object
> threshold is set to 1. Do you have any recommendations?
>
>
>
> Kevin
>
>
>
>
>
>
>
> *From:* Andy LoPresto [mailto:alopre...@apache.org]
> *Sent:* Thursday, April 27, 2017 4:16 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Back Pressure Object threshold not honored
>
>
>
> Hi Kevin,
>
>
>
> Sorry to hear you are having this issue. Can you please provide a
> screenshot of the connection properties in the configuration dialog? How
> quickly do those flowfiles get enqueued? I think there’s a chance if they
> are very small & the previous processor is very fast (i.e.
> RouteOnAttribute, SplitText) that it could enqueue a higher number before
> the back pressure check is executed.
>
>
>
> Andy LoPresto
>
> alopre...@apache.org
>
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
>
> On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven <kevin.verhoe...@ds-iq.com>
> wrote:
>
>
>
> I have an odd problem. I set the Back Pressure Object threshold on a link
> between two Processors to 1, but 200 flowfiles are passed to the queue
> before back pressure is honored. I need the back pressure to be set to a
> small number of flowfiles to keep the source from flooding the destination.
> Has anyone come across this problem before? I am running 12 instances of
> NiFi on version 1.1.1 on Ubuntu 14.04.
>
>
>
> Regards,
>
>
>
> Kevin
>
>
>
>
>
>
>
> --
>
> *Joe Percivall*
>
> e: joeperciv...@gmail.com
>
>

RE: Back Pressure Object threshold not honored

[Kevin Verhoeven]

Andrew, I will look at using the ControlRate processor as that might be a 
better alternative to throttling my workflow because I really need the outflow 
to be predictable.

Also, thanks Joe, I will be happy to test the change to UpdateAttribute – I 
have a test VM ready to go.

Thanks everyone for your input.

Regards,

Kevin

From: Andrew Grande [mailto:apere...@gmail.com]
Sent: Thursday, April 27, 2017 5:08 PM
To: users@nifi.apache.org
Subject: Re: Back Pressure Object threshold not honored


I would also suggest the ControlRate processor in front of a sensitive step. 
There's a chance data may accumulate and sent in burst after downtime, so it 
ensures a predictable outflow.

Andrew

On Thu, Apr 27, 2017, 8:03 PM Joe Percivall 
<joeperciv...@gmail.com<mailto:joeperciv...@gmail.com>> wrote:
Hello Kevin,

I believe there are two things at play here. The first is the processor being 
very fast and processing the FlowFiles before back pressure gets applied. The 
second is that in the current distribution, UpdateAttribute uses an old style 
of getting higher performance and grabs batches of 100 with each onTrigger[1]. 
Since back-pressure gets applied per onTrigger the UpdateAttribute will process 
at least 100 FlowFiles before it gets told to stop processing.

In the changes for 1.2.0 though I updated it to bring only 1 FlowFile in per 
onTrigger. So if you test this on a build of master then you should see more 
appropriate back-pressure application.

[1] 
https://github.com/apache/nifi/blob/rel/nifi-1.1.2/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338

Joe

On Thu, Apr 27, 2017 at 7:21 PM, Kevin Verhoeven 
<kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>> wrote:
Thank you for your help Andy. I think you are correct, the flowfiles are very 
small and the previous Processor is very fast – this might explain what is 
happening. I’ve enclosed screenshots of the connection properties and the 
workflow. In the screenshot I see 400 flowfiles were allowed through before 
back pressure was applied. The back pressure object threshold is set to 1. Do 
you have any recommendations?

Kevin

[cid:image001.png@01D2BF72.6461E580]

[cid:image002.png@01D2BF72.6461E580]

From: Andy LoPresto [mailto:alopre...@apache.org<mailto:alopre...@apache.org>]
Sent: Thursday, April 27, 2017 4:16 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Back Pressure Object threshold not honored

Hi Kevin,

Sorry to hear you are having this issue. Can you please provide a screenshot of 
the connection properties in the configuration dialog? How quickly do those 
flowfiles get enqueued? I think there’s a chance if they are very small & the 
previous processor is very fast (i.e. RouteOnAttribute, SplitText) that it 
could enqueue a higher number before the back pressure check is executed.

Andy LoPresto
alopre...@apache.org<mailto:alopre...@apache.org>
alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven 
<kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>> wrote:

I have an odd problem. I set the Back Pressure Object threshold on a link 
between two Processors to 1, but 200 flowfiles are passed to the queue before 
back pressure is honored. I need the back pressure to be set to a small number 
of flowfiles to keep the source from flooding the destination. Has anyone come 
across this problem before? I am running 12 instances of NiFi on version 1.1.1 
on Ubuntu 14.04.

Regards,

Kevin




--
Joe Percivall
e: joeperciv...@gmail.com<mailto:joeperciv...@gmail.com>

Re: Back Pressure Object threshold not honored

[Andrew Grande]

I would also suggest the ControlRate processor in front of a sensitive
step. There's a chance data may accumulate and sent in burst after
downtime, so it ensures a predictable outflow.

Andrew

On Thu, Apr 27, 2017, 8:03 PM Joe Percivall <joeperciv...@gmail.com> wrote:

> Hello Kevin,
>
> I believe there are two things at play here. The first is the processor
> being very fast and processing the FlowFiles before back pressure gets
> applied. The second is that in the current distribution, UpdateAttribute
> uses an old style of getting higher performance and grabs batches of 100
> with each onTrigger[1]. Since back-pressure gets applied per onTrigger the
> UpdateAttribute will process at least 100 FlowFiles before it gets told to
> stop processing.
>
> In the changes for 1.2.0 though I updated it to bring only 1 FlowFile in
> per onTrigger. So if you test this on a build of master then you should see
> more appropriate back-pressure application.
>
> [1]
> https://github.com/apache/nifi/blob/rel/nifi-1.1.2/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338
>
> Joe
>
> On Thu, Apr 27, 2017 at 7:21 PM, Kevin Verhoeven <
> kevin.verhoe...@ds-iq.com> wrote:
>
>> Thank you for your help Andy. I think you are correct, the flowfiles are
>> very small and the previous Processor is very fast – this might explain
>> what is happening. I’ve enclosed screenshots of the connection properties
>> and the workflow. In the screenshot I see 400 flowfiles were allowed
>> through before back pressure was applied. The back pressure object
>> threshold is set to 1. Do you have any recommendations?
>>
>>
>>
>> Kevin
>>
>>
>>
>>
>>
>>
>>
>> *From:* Andy LoPresto [mailto:alopre...@apache.org]
>> *Sent:* Thursday, April 27, 2017 4:16 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Back Pressure Object threshold not honored
>>
>>
>>
>> Hi Kevin,
>>
>>
>>
>> Sorry to hear you are having this issue. Can you please provide a
>> screenshot of the connection properties in the configuration dialog? How
>> quickly do those flowfiles get enqueued? I think there’s a chance if they
>> are very small & the previous processor is very fast (i.e.
>> RouteOnAttribute, SplitText) that it could enqueue a higher number before
>> the back pressure check is executed.
>>
>>
>>
>> Andy LoPresto
>>
>> alopre...@apache.org
>>
>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>>
>>
>> On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven <kevin.verhoe...@ds-iq.com>
>> wrote:
>>
>>
>>
>> I have an odd problem. I set the Back Pressure Object threshold on a link
>> between two Processors to 1, but 200 flowfiles are passed to the queue
>> before back pressure is honored. I need the back pressure to be set to a
>> small number of flowfiles to keep the source from flooding the destination.
>> Has anyone come across this problem before? I am running 12 instances of
>> NiFi on version 1.1.1 on Ubuntu 14.04.
>>
>>
>>
>> Regards,
>>
>>
>>
>> Kevin
>>
>>
>>
>
>
>
> --
> *Joe Percivall*
> e: joeperciv...@gmail.com
>

Re: Back Pressure Object threshold not honored

[Joe Percivall]

Hello Kevin,

I believe there are two things at play here. The first is the processor
being very fast and processing the FlowFiles before back pressure gets
applied. The second is that in the current distribution, UpdateAttribute
uses an old style of getting higher performance and grabs batches of 100
with each onTrigger[1]. Since back-pressure gets applied per onTrigger the
UpdateAttribute will process at least 100 FlowFiles before it gets told to
stop processing.

In the changes for 1.2.0 though I updated it to bring only 1 FlowFile in
per onTrigger. So if you test this on a build of master then you should see
more appropriate back-pressure application.

[1]
https://github.com/apache/nifi/blob/rel/nifi-1.1.2/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338

Joe

On Thu, Apr 27, 2017 at 7:21 PM, Kevin Verhoeven <kevin.verhoe...@ds-iq.com>
wrote:

> Thank you for your help Andy. I think you are correct, the flowfiles are
> very small and the previous Processor is very fast – this might explain
> what is happening. I’ve enclosed screenshots of the connection properties
> and the workflow. In the screenshot I see 400 flowfiles were allowed
> through before back pressure was applied. The back pressure object
> threshold is set to 1. Do you have any recommendations?
>
>
>
> Kevin
>
>
>
>
>
>
>
> *From:* Andy LoPresto [mailto:alopre...@apache.org]
> *Sent:* Thursday, April 27, 2017 4:16 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Back Pressure Object threshold not honored
>
>
>
> Hi Kevin,
>
>
>
> Sorry to hear you are having this issue. Can you please provide a
> screenshot of the connection properties in the configuration dialog? How
> quickly do those flowfiles get enqueued? I think there’s a chance if they
> are very small & the previous processor is very fast (i.e.
> RouteOnAttribute, SplitText) that it could enqueue a higher number before
> the back pressure check is executed.
>
>
>
> Andy LoPresto
>
> alopre...@apache.org
>
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
>
> On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven <kevin.verhoe...@ds-iq.com>
> wrote:
>
>
>
> I have an odd problem. I set the Back Pressure Object threshold on a link
> between two Processors to 1, but 200 flowfiles are passed to the queue
> before back pressure is honored. I need the back pressure to be set to a
> small number of flowfiles to keep the source from flooding the destination.
> Has anyone come across this problem before? I am running 12 instances of
> NiFi on version 1.1.1 on Ubuntu 14.04.
>
>
>
> Regards,
>
>
>
> Kevin
>
>
>



-- 
*Joe Percivall*
e: joeperciv...@gmail.com

RE: Back Pressure Object threshold not honored

[Kevin Verhoeven]

Thank you for your help Andy. I think you are correct, the flowfiles are very 
small and the previous Processor is very fast – this might explain what is 
happening. I’ve enclosed screenshots of the connection properties and the 
workflow. In the screenshot I see 400 flowfiles were allowed through before 
back pressure was applied. The back pressure object threshold is set to 1. Do 
you have any recommendations?

Kevin

[cid:image001.png@01D2BF72.6461E580]

[cid:image002.png@01D2BF72.6461E580]

From: Andy LoPresto [mailto:alopre...@apache.org]
Sent: Thursday, April 27, 2017 4:16 PM
To: users@nifi.apache.org
Subject: Re: Back Pressure Object threshold not honored

Hi Kevin,

Sorry to hear you are having this issue. Can you please provide a screenshot of 
the connection properties in the configuration dialog? How quickly do those 
flowfiles get enqueued? I think there’s a chance if they are very small & the 
previous processor is very fast (i.e. RouteOnAttribute, SplitText) that it 
could enqueue a higher number before the back pressure check is executed.

Andy LoPresto
alopre...@apache.org<mailto:alopre...@apache.org>
alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven 
<kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>> wrote:

I have an odd problem. I set the Back Pressure Object threshold on a link 
between two Processors to 1, but 200 flowfiles are passed to the queue before 
back pressure is honored. I need the back pressure to be set to a small number 
of flowfiles to keep the source from flooding the destination. Has anyone come 
across this problem before? I am running 12 instances of NiFi on version 1.1.1 
on Ubuntu 14.04.

Regards,

Kevin

Re: Back Pressure Object threshold not honored

[Andy LoPresto]

Hi Kevin,

Sorry to hear you are having this issue. Can you please provide a screenshot of 
the connection properties in the configuration dialog? How quickly do those 
flowfiles get enqueued? I think there’s a chance if they are very small & the 
previous processor is very fast (i.e. RouteOnAttribute, SplitText) that it 
could enqueue a higher number before the back pressure check is executed.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven  
> wrote:
> 
> I have an odd problem. I set the Back Pressure Object threshold on a link 
> between two Processors to 1, but 200 flowfiles are passed to the queue before 
> back pressure is honored. I need the back pressure to be set to a small 
> number of flowfiles to keep the source from flooding the destination. Has 
> anyone come across this problem before? I am running 12 instances of NiFi on 
> version 1.1.1 on Ubuntu 14.04.
> 
> Regards,
> 
> Kevin



signature.asc
Description: Message signed with OpenPGP using GPGMail

Back Pressure Object threshold not honored

[Kevin Verhoeven]

I have an odd problem. I set the Back Pressure Object threshold on a link 
between two Processors to 1, but 200 flowfiles are passed to the queue before 
back pressure is honored. I need the back pressure to be set to a small number 
of flowfiles to keep the source from flooding the destination. Has anyone come 
across this problem before? I am running 12 instances of NiFi on version 1.1.1 
on Ubuntu 14.04.

Regards,

Kevin