[
https://issues.apache.org/jira/browse/NIFI-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sönke Liebau reassigned NIFI-16558:
-----------------------------------
Assignee: Sönke Liebau
> Nodes accept Reconnection Requests from nodes that are not the elected
> Cluster Coordinator
> ------------------------------------------------------------------------------------------
>
> Key: NIFI-16558
> URL: https://issues.apache.org/jira/browse/NIFI-16558
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Sönke Liebau
> Assignee: Sönke Liebau
> Priority: Minor
>
> Could also be argued to be a bug .. not sure tbh..
>
> A node handles any RECONNECTION_REQUEST it receives. It doesn't check that
> the sender is the currently elected coordinator, or that it is connected to a
> different coordinator already. It leaves its current cluster connection,
> loads whatever flow came with the request and replaces its node status table
> with the one from the request:
>
> The request carries the sender's instanceId, but nothing compares it to
> anything. Combined with the sender side not cancelling stale reconnect
> threads (linked issue), a former coordinator can push its flow and its
> outdated view of the cluster onto a node that is happily connected to the new
> coordinator. If that former coordinator is shutting down, the flow has all
> processors stopped, which is how we ended up with a node with stopped
> processors after a rolling restart.
> h3. Reproduction
> Claude orchestrated this on our 3 node test cluster (2.7.2, Kubernetes leader
> election). I followed along in the logs and checked the cluster state before
> and after.
> * node-0: the target
> * node-1: coordinator
> * node-2: just the third node
> Using iptables in the pods' network namespaces, we cut protocol port traffic
> between node-0 and node-1 until node-1 disconnected node-0 for lack of
> heartbeat. Then we let one heartbeat through so node-1 started its Reconnect
> thread, and switched node-0's inbound rule to DROP so the next attempt hangs
> in connect (see NIFI-16547).
>
> After that node-1 was cut off from the API server (so it loses the lease) and
> from node-2 on the protocol port (so it doesn't learn about status changes,
> same as a coordinator that is shutting down).
>
> {code:java}
> node-1 13:24:12,723 [Reconnect nifi-node-default-0...] Problem encountered
> issuing reconnection request to node nifi-node-default-0...
> node-1 13:24:34,890 FlowController This node is no longer the elected Active
> Cluster Coordinator
> node-2 13:24:36,889 [Heartbeat Monitor Thread-1] nifi-node-default-0... is
> now connected
> {code}
>
> From here node-0 was connected to node-2 (the new coordinator), the lease was
> held by node-2 and node-1 was disconnected from node-2's cluster. Four
> minutes later we removed the DROP rule. node-1's Reconnect thread was still
> alive (attempts at 13:26:30 and 13:28:49 timed out or failed) and got through:
>
> {code:java}
> node-1 13:28:54,588 [Reconnect nifi-node-default-0...] Successfully requested
> that nifi-node-default-0... join the cluster
> node-0 13:28:54,586 [Reconnect to Cluster] StandardFlowService Processing
> reconnection request from cluster coordinator.
> node-0 13:28:54,587 [Reconnect to Cluster] NodeClusterCoordinator Resetting
> cluster node statuses from {node-0 CONNECTED updateId=179, node-1
> DISCONNECTED (Lack of Heartbeat) updateId=180,node-2 CONNECTED updateId=173}
> to {node-0 CONNECTING updateId=179,node-2 CONNECTED updateId=173, node-1
> CONNECTED updateId=168}
> node-0 13:28:54,664 [Reconnect to Cluster] StandardFlowService Node
> reconnected.
> {code}
>
> (status lines shortened) node-0 replaced a correct status table with node-1's
> 4 minute old one, which has the disconnected node-1 as CONNECTED, and loaded
> node-1's flow. Processors on node-1 were running, so nothing visibly broke in
> this run.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
