markap14 commented on a change in pull request #4045: NIFI-7117: When
SocketLoadBalancedFlowFileQueue creates its array of …
URL: https://github.com/apache/nifi/pull/4045#discussion_r377809799
##########
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java
##########
@@ -151,15 +151,19 @@ public SocketLoadBalancedFlowFileQueue(final String
identifier, final Connection
// that is not the local node identifier. If the Local Node
Identifier is not yet known, that's okay. When it becomes known,
// the queuePartitions array will be recreated with the
appropriate partitions.
final List<QueuePartition> partitionList = new ArrayList<>();
- partitionList.add(localPartition);
final NodeIdentifier localNodeId =
clusterCoordinator.getLocalNodeIdentifier();
for (final NodeIdentifier nodeId : sortedNodeIdentifiers) {
if (nodeId.equals(localNodeId)) {
- continue;
+ partitionList.add(localPartition);
+ } else {
+ partitionList.add(createRemotePartition(nodeId));
}
+ }
- partitionList.add(createRemotePartition(nodeId));
+ // Ensure that our list of queue partitions always contains the
local partition.
+ if (!partitionList.contains(localPartition)) {
Review comment:
You're correct. If local node id ends up being null, we will add
localPartition to the end of this, which means that this node will have the
order wrong. As the code comment above explains, though, that's okay. The order
will only be wrong until the local node identifier is determined. This will
generally take milliseconds. Perhaps a few seconds. Then the ordering will be
fixed in the `onLocalNodeIdentifierSet` method. In the meantime, it may end up
sending the data to the wrong node, and the receiving node would then take that
data and re-send it to who it believes to be the correct node in the cluster.
It is inefficient, but should only occur for a very short period of time and
only when the local node id is null (and this happens rarely, as it's basically
a race condition that can happen when the node is added to the cluster for the
first time. The next time the cluster is restarted, the local node id should be
known from local state.)
The problem previously was that the localPartition was *always* added as the
first entry, even when the local node identifier was already known. As a
result, it didn't get re-sorted. Now, it will be added only when the local nod
id is null and therefore it will be re-sorted when the local node id is known.
Sorry for the huge explanation. I hope it all makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
