[
https://issues.apache.org/jira/browse/NIFI-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026192#comment-17026192
]
Mark Payne commented on NIFI-7076:
----------------------------------
This looks to have been caused by NIFI-6787. With the introduction of Load
Balanced Connections, the internal structure of the flowfile queue is such that
it has N different partitions - 1 for the local node and one for each node in
the cluster. Backpressure should be applied if the number of FlowFIles queued
is >= the backpressure threshold. The solution provided for NIFI-6787 updated
the logic so that backpressure will instead be engaged only if *all* partitions
in and of themselves have reached the backpressure threshold.
There were two problems with this approach that were not previously realized.
Firstly, if a connection is not Load Balanced, data only goes to the 'local
partition' and so there always (assuming 2 or more nodes in the cluster) will
be an empty partition, so backpressure never gets applied. That's the most
critical concern, which is what is outlined here.
Secondly, even with the above problem addressed, we still would have a
situation where in order for backpressure to be applied, if there are N nodes
in the cluster, we must reach N times the configured backpressure. For example,
if backpressure threshold is at 10,000 FlowFiles and there are 3 nodes, we have
to reach 30,000 FlowFiles in the queue before backpressure is applied on that
node.
The best course of action is to revert NIFI-6787, and then file another Jira
for looking at alternate methods to improve the situation that it was intended
to address.
> Back Pressure Not Being Applied In Cluster Mode
> -----------------------------------------------
>
> Key: NIFI-7076
> URL: https://issues.apache.org/jira/browse/NIFI-7076
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.11.0
> Reporter: Shawn Weeks
> Priority: Critical
> Fix For: 1.12.0, 1.11.1
>
> Attachments: Bug_Example.xml, Screen Shot 2020-01-29 at 12.49.17
> PM.png, Screen Shot 2020-01-29 at 12.49.55 PM.png
>
>
> In the attached example back pressure does not get applied if executed on a
> NiFi cluster. Single node works fine. I've reproduced this locally on a
> Redhat 7 based cluster with embedded zookeeper and a Kubernetes based cluster
> running external zookeeper.
> This has been reproduced both with the embedded zookeeper with 1.11 and
> external docker based zookeeper from
> docker.io/bitnami/zookeeper:3.5.6-debian-10-r0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
