[
https://issues.apache.org/jira/browse/NIFI-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643237#comment-16643237
]
ASF GitHub Bot commented on NIFI-5663:
--------------------------------------
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/3048
LGTM +1 merging. Thank you @markap14 for fixing this!
> FlowFile load balancing keeps re-partitioning
> ---------------------------------------------
>
> Key: NIFI-5663
> URL: https://issues.apache.org/jira/browse/NIFI-5663
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.8.0
> Reporter: Koji Kawamura
> Assignee: Mark Payne
> Priority: Critical
>
> Scenario
> # Start a brand-new cluster with only 1 node (nifi0). Using existing
> multi-node clusters don't reproduce the issue.
> # Create GenerateFlowFile -> LogAttribute
> # Then set 'Partition by attribute' LB strategy at the connection
> # Add 2nd node, nifi1
> # Generate some FlowFiles. Then load-balance activity never finishes.
> With a 2-node cluster, for some reason, each node ended up having different
> queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug
> logs, I found each node has followings:
> * nifi0
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b,
> nodeId=nifi1.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b]
> * nifi1
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b,
> nodeId=nifi0.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b]
> Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
> received FlowFiles between each other in case the calculated attribute value
> hash points to queuePartitions[0]. Following log is written endlessly:
> {code:java}
> 2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
> o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
> from Peer: ...offset=7452,
> length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
> FlowFiles to ensure proper balancing across the cluster.
> {code}
> SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
> cluster topology change using ClusterTopologyEventListener.
> SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log
> shows the array was empty when the 2nd node (nifi1) is added:
> {code:java}
> ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
> Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
> Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from
> [] to [nifi1.example.com:8080]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
