Vincenzo Lombardo created NIFI-16542:
----------------------------------------
Summary: Removing a single cluster state key disconnects nodes due
to non-idempotent replicated validation
Key: NIFI-16542
URL: https://issues.apache.org/jira/browse/NIFI-16542
Project: Apache NiFi
Issue Type: Bug
Reporter: Vincenzo Lombardo
When a single key is removed from a component's cluster state (POST
/processors/\{id}/state/clear-requests with a ComponentStateDTO, the selective
key drop added in NIFI-14900), the request is replicated to all nodes, but the
cluster state itself is shared (e.g. ZooKeeper).
StandardComponentStateDAO.clearState() validates the request on each node with
hasExactlyOneKeyRemoved(currentClusterState, newClusterState) before calling
setState(). The first node to process the request validates against the
original state, writes the new state and returns 200. Any node that processes
the request afterwards reads the already-updated cluster state, where current
equals new, so the validation fails with IllegalStateException ("Unable to
remove a state key for component ... Exactly one key removal is supported.")
and that node returns 409.
Because the node responses differ, the cluster coordinator requests the failing
node to reconnect:
WARN NodeClusterCoordinator The following nodes failed to process URI
/nifi-api/processors/<id>/state/clear-requests '[localhost:5672]'. Requesting
each node reconnect to cluster.
In the node request logs the same clear-requests call shows 200 on one node and
409 on the other.
This is visible in the system tests: since NIFI-16494 added a second
runProcessorOnce() at the end of
ClusterStateKeyDropIT.testCanDropSpecificStateKey, the test fails frequently on
main (macOS and Ubuntu, Java 21 and 25) with:
NiFiClientException: Error starting Processor: Cluster is unable to service
request to change flow: Node localhost:5672 is currently connecting
The retry in AbstractStateKeyDropIT only covers "not connected" messages, but
the underlying cause is the node disconnection described above, so extending
the retry would only hide the problem. Before NIFI-16494 the race was already
present but went unnoticed because the test ended right after verifying the
state (NIFI-15763 likely addressed an earlier symptom of the same issue).
Example failing runs on main, with troubleshooting-logs artifacts containing
the node logs:
https://github.com/apache/nifi/actions/runs/41653533114
https://github.com/apache/nifi/actions/runs/41534185546
https://github.com/apache/nifi/actions/runs/41100346795
Proposed fix: make the selective cluster state key removal idempotent across
replicated requests. If the current cluster state already equals the requested
state, treat the request as already applied by another node instead of
throwing. Add a unit test covering the second-node case.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
