Sönke Liebau created NIFI-16557:
-----------------------------------
Summary: Cluster Coordinator that is shutting down can wrongfully
send a Reconnection Request containing a flow with all processors stopped
Key: NIFI-16557
URL: https://issues.apache.org/jira/browse/NIFI-16557
Project: Apache NiFi
Issue Type: Bug
Components: Core UI
Affects Versions: 2.7.2
Environment: NiFi on Kubernetes - should not be relevant for this bug
though
Reporter: Sönke Liebau
h3. Symptoms
We hit this during a rolling restart of a 3 node cluster on Kubernetes
(Kubernetes leader election). Afterwards one node had all processors stopped
while the other two were running, and the merged view in the UI showed the
processors as stopped. `autoResumeState` was true everywhere. The stopped node
had joined the new coordinator fine and started its processors. A minute later
it got a RECONNECTION_REQUEST whose flow had every processor stopped, applied
it ("will not be restarted because the proposed flow indicates that they are
not running") and stayed that way. The new coordinator never sent that request.
Going by the node status table attached to it, it came from the previous
coordinator, which was shutting down at the time. Its logs were lost with the
pod, so for the original incident that part is conjecture.
h3. Code
`requestNodeConnect` hands off to `requestReconnectionAsynchronously`, which
[retries 10 times with 5 seconds in
between|https://github.com/apache/nifi/blob/rel/nifi-2.12.0/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java#L1082-L1129]
and
[serializes|https://github.com/apache/nifi/blob/rel/nifi-2.12.0/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java#L1103-L1105]
the local flow again on every attempt:
The only guard is the local view of the target being CONNECTING. The thread
never checks closed (set in shutdown() but never read by the reconnect thread)
or whether this node is still the active coordinator. Nothing interrupts it on
shutdown either.
`StandardFlowService.stop()` shuts down the coordinator and the protocol
listener first and then calls FlowController.shutdown(), which [stops the root
group|https://github.com/apache/nifi/blob/rel/nifi-2.12.0/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L1895-L1898]
and sets every processor's desired state to STOPPED. That gets serialized as
ENABLED. The sender side stays functional the whole time, so any reconnect
attempt that fires in that window ships the stopped flow.
h3. Prior Art
NIFI-15519 fixed the same window for saving flow.json.gz but not for flows sent
to other nodes.
This may potentially be related to NIFI-16547 in that the hangs introduced by
that bug can of course make this seem worse than it actually is. We have not
fully investigated the dependencies to the bitter end yet. But at least we came
across it during the same investigation that also turned up NIFI-16547.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
