[
https://issues.apache.org/jira/browse/NIFI-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898374#comment-15898374
]
Koji Kawamura commented on NIFI-3559:
-------------------------------------
[~msclarke] I've been working on NIFI-1202, which adds S2S batch configuration
to remote port. It's mostly done as I summarized status on [Github PR
1306|https://github.com/apache/nifi/pull/1306].
Same as NIFI-2987, do you think NIFI-1202 will complete this improvement
request, or is there any additional enhancement required?
> Improve S2S load-balancing
> --------------------------
>
> Key: NIFI-3559
> URL: https://issues.apache.org/jira/browse/NIFI-3559
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.1.1
> Reporter: Matthew Clarke
> Assignee: Koji Kawamura
>
> The current implementation of S2S sends data continuously to the destination
> NiFi node for 0.5 seconds before closing the connection and opening a new
> connection to another node.
> When the source FlowFile are all very small (0 byte in case of list based
> processors), the entire queue can end up getting sent to only one of the
> target NiFi cluster nodes.
> Another common use case for S2S is to have a RPG pointed back at same cluster
> where the RPG was added. Since FlowFiles are likely to transfer to the same
> node where the data originates (Think Primary node data redistribution within
> a cluster) much faster then transfers to other nodes, the primary node is
> likely to always end up with more FlowFiles then any other node.
> There needs to be an additional load-balancing strategy that compliments the
> existing 0.5 second to improve upon the load-balancing in such cases. The
> RPG know how many target nodes there are and how many FlowFiles exist in the
> queue at run time, so perhaps using that info to more even split the queue
> amongst all nodes smartly would help.
> This is related to existing Jira: NiFI-2987
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
