Hey Paul, Looks like you’re running into https://issues.apache.org/jira/browse/NIFI-10246, which was addressed in 1.17.0 When NiFi gets to that point, the service is either in an ENABLING state (even though it says it’s enabled, it may just be enabling) or it’s decided that it needs to disable the service in order to ensure that the flow is in sync with the cluster.
Unfortunately, there was an issue where it would not properly wait for the service to be completely DISABLED. It would asynchronously disable the service and then go ahead without waiting for that action to complete. That produced the issue that you’re seeing in the stack trace. Thanks -Mark On Jan 12, 2023, at 11:36 AM, Paul Riddle <[email protected]> wrote: Good Morning NiFi Folks! NiFi Version: 1.16.3 Issue: Enabling DistributedMapCacheClient is causing clustered nodes to not be able to startup or rejoin cluster after NiFi stop/restart. Scenario: I am using a DetectDuplicate processor on a 3 node cluster. I have a DistributedMapCacheServer controller service running on one of the 3 nodes. It enables and works fine. I also have a DistributeMapCacheClient controller service configured. It appears to enable and work fine. I run data through DetectDuplicate. It finds duplicates as I would expect it to do. An issue presents itself, however, when NiFi is restarted for any reason. If, and only if, the DistributedMapCacheClient service (I verified each other service) is set to ENABLED when NiFi initially starts up or tries to restart, NiFi fails to start up and I receive a stack trace in the logs that says: ----------------------------------------------------------------------------- “2023-01-12 16:06:59,386 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption. org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption. at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1061) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:524) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1086) at org.apache.nifi.NiFi.<init>(NiFi.java:170) at org.apache.nifi.NiFi.<init>(NiFi.java:82) at org.apache.nifi.NiFi.main(NiFi.java:330) Caused by: org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalStateException: Cannot modify Controller Service configuration because it is currently enabled. Please disable the Controller Service firs t. at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:370) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:188) at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524) at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:107) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:819) at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1030)” ----------------------------------------------------------------------------- At first glance it seems that NiFi is trying to modify something within the DistributedMapCacheClient service on startup after it is in an ENABLED state for some reason. It appears to be an order of operations issue. Is this a known bug? Is there some sort of workaround other than disabling the controller service every time before I shut NiFi down? Thanks for the Help! Paul
