Koji Kawamura created NIFI-5663:
-----------------------------------
Summary: FlowFile load balancing keeps re-partitioning
Key: NIFI-5663
URL: https://issues.apache.org/jira/browse/NIFI-5663
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.8.0
Reporter: Koji Kawamura
Scenario
# Start a brand-new cluster with only 1 node. nifi0
# Create GenerateFlowFile -> LogAttribute
# Then set 'Partition by attribute' LB strategy at the connection
# Add 2nd node, nifi1
# Generate some FlowFiles. Then load-balance activity never finishes.
With a 2-node cluster, for some reason, each node ended up having different
queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug logs,
I found each node has followings:
* nifi0
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b,
nodeId=nifi1.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b]
* nifi1
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b,
nodeId=nifi0.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b]
Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
received FlowFiles between each other in case the calculated attribute value
hash points to queuePartitions[0]. Following log is written endlessly:
{code:java}
2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
from Peer: ...offset=7452,
length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
FlowFiles to ensure proper balancing across the cluster.
{code}
SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
cluster topology change using ClusterTopologyEventListener.
SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log shows
the array was empty when the 2nd node (nifi1) is added:
{code:java}
ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from []
to [nifi1.example.com:8080]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
