[
https://issues.apache.org/jira/browse/NIFI-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014519#comment-18014519
]
Praveen Dharam commented on NIFI-14841:
---------------------------------------
We have several custom NiFi processors in our workflow. Here’s the step-by-step
flow of what happens:
We have a custom Processor P1: This processor receives flow files containing
comma-separated records, with each file holding multiple records. It processes
all files in each batch, splits the records into multiple output flow files
based on certain fields that are concatenated into a string, and assigns that
string as a flow file attribute.
We run this in a 3-node NiFi cluster, configured with the Partition by
Attribute load balancing strategy to distribute these files.
Custom Processor P2: This processor consumes the output of P1, where the
expectation is that each node receives flow files with a unique attribute value.
In NiFi 2.0, using the same input files, we observed that the load was
distributed fairly evenly across all three nodes (with only minor percentage
differences). However, in NiFi 2.4, using the same input files, the majority of
the load is sent to a single node, while the other two nodes receive
significantly less.
> In NiFi 2.4, the load distribution is uneven, whereas it worked correctly in
> NiFi 2.0
> -------------------------------------------------------------------------------------
>
> Key: NIFI-14841
> URL: https://issues.apache.org/jira/browse/NIFI-14841
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Praveen Dharam
> Priority: Major
>
> In NiFi 2.4, we are experiencing uneven load distribution across nodes when
> the load balancing strategy is set to Partition by Attribute.
> The primary issue is that the majority of the load is being routed to a
> single node out of the three in the cluster.
> This imbalance has significantly reduced our throughput during volume
> testing—an issue that was not present in NiFi 2.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
