[
https://issues.apache.org/jira/browse/NIFI-16542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincenzo Lombardo updated NIFI-16542:
-------------------------------------
Component/s: Core Framework
> Removing a single cluster state key disconnects nodes due to non-idempotent
> replicated validation
> -------------------------------------------------------------------------------------------------
>
> Key: NIFI-16542
> URL: https://issues.apache.org/jira/browse/NIFI-16542
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.12.0
> Reporter: Vincenzo Lombardo
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When a single key is removed from a component's cluster state (POST
> /processors/\{id}/state/clear-requests with a ComponentStateDTO, the
> selective key drop added in NIFI-14900), the request is replicated to all
> nodes, but the cluster state itself is shared (e.g. ZooKeeper).
> StandardComponentStateDAO.clearState() validates the request on each node
> with hasExactlyOneKeyRemoved(currentClusterState, newClusterState) before
> calling setState(). The first node to process the request validates against
> the original state, writes the new state and returns 200. Any node that
> processes the request afterwards reads the already-updated cluster state,
> where current equals new, so the validation fails with IllegalStateException
> ("Unable to remove a state key for component ... Exactly one key removal is
> supported.") and that node returns 409.
> Because the node responses differ, the cluster coordinator requests the
> failing node to reconnect:
> WARN NodeClusterCoordinator The following nodes failed to process URI
> /nifi-api/processors/<id>/state/clear-requests '[localhost:5672]'.
> Requesting each node reconnect to cluster.
> In the node request logs the same clear-requests call shows 200 on one node
> and 409 on the other.
> This is visible in the system tests: since NIFI-16494 added a second
> runProcessorOnce() at the end of
> ClusterStateKeyDropIT.testCanDropSpecificStateKey, the test fails frequently
> on main (macOS and Ubuntu, Java 21 and 25) with:
> NiFiClientException: Error starting Processor: Cluster is unable to service
> request to change flow: Node localhost:5672 is currently connecting
> The retry in AbstractStateKeyDropIT only covers "not connected" messages, but
> the underlying cause is the node disconnection described above, so extending
> the retry would only hide the problem. Before NIFI-16494 the race was already
> present but went unnoticed because the test ended right after verifying the
> state (NIFI-15763 likely addressed an earlier symptom of the same issue).
> Example failing runs on main, with troubleshooting-logs artifacts containing
> the node logs:
> https://github.com/apache/nifi/actions/runs/41653533114
> https://github.com/apache/nifi/actions/runs/41534185546
> https://github.com/apache/nifi/actions/runs/41100346795
> Proposed fix: make the selective cluster state key removal idempotent across
> replicated requests. If the current cluster state already equals the
> requested state, treat the request as already applied by another node instead
> of throwing. Add a unit test covering the second-node case.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
