Sönke Liebau created NIFI-16558:
-----------------------------------
Summary: Nodes accept Reconnection Requests from nodes that are
not the elected Cluster Coordinator
Key: NIFI-16558
URL: https://issues.apache.org/jira/browse/NIFI-16558
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: Sönke Liebau
Could also be argued to be a bug .. not sure tbh..
A node handles any RECONNECTION_REQUEST it receives. It doesn't check that the
sender is the currently elected coordinator, or that it is connected to a
different coordinator already. It leaves its current cluster connection, loads
whatever flow came with the request and replaces its node status table with the
one from the request:
The request carries the sender's instanceId, but nothing compares it to
anything. Combined with the sender side not cancelling stale reconnect threads
(linked issue), a former coordinator can push its flow and its outdated view of
the cluster onto a node that is happily connected to the new coordinator. If
that former coordinator is shutting down, the flow has all processors stopped,
which is how we ended up with a node with stopped processors after a rolling
restart.
h3. Reproduction
Claude orchestrated this on our 3 node test cluster (2.7.2, Kubernetes leader
election). I followed along in the logs and checked the cluster state before
and after.
* node-0: the target
* node-1: coordinator
* node-2: just the third node
Using iptables in the pods' network namespaces, we cut protocol port traffic
between node-0 and node-1 until node-1 disconnected node-0 for lack of
heartbeat. Then we let one heartbeat through so node-1 started its Reconnect
thread, and switched node-0's inbound rule to DROP so the next attempt hangs in
connect (see NIFI-16547).
After that node-1 was cut off from the API server (so it loses the lease) and
from node-2 on the protocol port (so it doesn't learn about status changes,
same as a coordinator that is shutting down).
{code:java}
node-1 13:24:12,723 [Reconnect nifi-node-default-0...] Problem encountered
issuing reconnection request to node nifi-node-default-0...
node-1 13:24:34,890 FlowController This node is no longer the elected Active
Cluster Coordinator
node-2 13:24:36,889 [Heartbeat Monitor Thread-1] nifi-node-default-0... is now
connected
{code}
>From here node-0 was connected to node-2 (the new coordinator), the lease was
>held by node-2 and node-1 was disconnected from node-2's cluster. Four minutes
>later we removed the DROP rule. node-1's Reconnect thread was still alive
>(attempts at 13:26:30 and 13:28:49 timed out or failed) and got through:
{code:java}
node-1 13:28:54,588 [Reconnect nifi-node-default-0...] Successfully requested
that nifi-node-default-0... join the cluster
node-0 13:28:54,586 [Reconnect to Cluster] StandardFlowService Processing
reconnection request from cluster coordinator.
node-0 13:28:54,587 [Reconnect to Cluster] NodeClusterCoordinator Resetting
cluster node statuses from {node-0 CONNECTED updateId=179, node-1 DISCONNECTED
(Lack of Heartbeat) updateId=180,node-2 CONNECTED updateId=173} to {node-0
CONNECTING updateId=179,node-2 CONNECTED updateId=173, node-1 CONNECTED
updateId=168}
node-0 13:28:54,664 [Reconnect to Cluster] StandardFlowService Node reconnected.
{code}
(status lines shortened) node-0 replaced a correct status table with node-1's 4
minute old one, which has the disconnected node-1 as CONNECTED, and loaded
node-1's flow. Processors on node-1 were running, so nothing visibly broke in
this run.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
