[
https://issues.apache.org/jira/browse/NIFI-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sönke Liebau reassigned NIFI-16557:
-----------------------------------
Assignee: Sönke Liebau
> Cluster Coordinator that is shutting down can wrongfully send a Reconnection
> Request containing a flow with all processors stopped
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16557
> URL: https://issues.apache.org/jira/browse/NIFI-16557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 2.7.2
> Environment: NiFi on Kubernetes - should not be relevant for this bug
> though
> Reporter: Sönke Liebau
> Assignee: Sönke Liebau
> Priority: Minor
>
> h3. Symptoms
> We hit this during a rolling restart of a 3 node cluster on Kubernetes
> (Kubernetes leader election). Afterwards one node had all processors stopped
> while the other two were running, and the merged view in the UI showed the
> processors as stopped. `autoResumeState` was true everywhere. The stopped
> node had joined the new coordinator fine and started its processors. A minute
> later it got a RECONNECTION_REQUEST whose flow had every processor stopped,
> applied it ("will not be restarted because the proposed flow indicates that
> they are not running") and stayed that way. The new coordinator never sent
> that request. Going by the node status table attached to it, it came from the
> previous coordinator, which was shutting down at the time. Its logs were lost
> with the pod, so for the original incident that part is conjecture.
> h3. Code
> `requestNodeConnect` hands off to `requestReconnectionAsynchronously`, which
> [retries 10 times with 5 seconds in
> between|https://github.com/apache/nifi/blob/rel/nifi-2.12.0/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java#L1082-L1129]
> and
> [serializes|https://github.com/apache/nifi/blob/rel/nifi-2.12.0/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java#L1103-L1105]
> the local flow again on every attempt:
> The only guard is the local view of the target being CONNECTING. The thread
> never checks closed (set in shutdown() but never read by the reconnect
> thread) or whether this node is still the active coordinator. Nothing
> interrupts it on shutdown either.
>
> `StandardFlowService.stop()` shuts down the coordinator and the protocol
> listener first and then calls FlowController.shutdown(), which [stops the
> root
> group|https://github.com/apache/nifi/blob/rel/nifi-2.12.0/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L1895-L1898]
> and sets every processor's desired state to STOPPED. That gets serialized as
> ENABLED. The sender side stays functional the whole time, so any reconnect
> attempt that fires in that window ships the stopped flow.
> h3. Prior Art
> NIFI-15519 fixed the same window for saving flow.json.gz but not for flows
> sent to other nodes.
>
> This may potentially be related to NIFI-16547 in that the hangs introduced by
> that bug can of course make this seem worse than it actually is. We have not
> fully investigated the dependencies to the bitter end yet. But at least we
> came across it during the same investigation that also turned up NIFI-16547.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
