[
https://issues.apache.org/jira/browse/NIFI-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17923857#comment-17923857
]
David Handermann commented on NIFI-14236:
-----------------------------------------
Thanks for highlighting this issue [~markap14].
Reviewing the Google Guava implementation, the consistent hash method is based
on an internal utility class:
[https://github.com/google/guava/blob/453549c87a83412ba3521201fe85d77ffe7d495d/guava/src/com/google/common/hash/Hashing.java#L837C30-L837C57]
With Google Guava being licensed under Apache License Version 2, it seems like
that particular utility could be brought in with appropriate attribution.
> Load Balance using Partition by Attribute results in poor distribution
> ----------------------------------------------------------------------
>
> Key: NIFI-14236
> URL: https://issues.apache.org/jira/browse/NIFI-14236
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Priority: Major
>
> NIFI-9638 aimed to eliminate unnecessary dependencies on the Google Guava
> library. However, in doing so, it changed the hashing algorithm from
> Consistent Hashing to a much simpler hashing algorithm that results in poor
> distribution of data.
> Consistent Hashing was specifically chosen for this use case because it
> provides excellent distribution of data across a given spectrum of bins, and
> also is designed such that if the number of bins (in this case number of NiFi
> nodes) changes, the number of elements that need to be re-distributed (in
> this case the number of FlowFiles) is minimal.
> We need to revert back to Consistent Hashing in order to provide better
> distribution of data across the cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
