Mark Payne created NIFI-16289:
---------------------------------
Summary: Fix flaky Couchbase processor integration-test bucket
readiness
Key: NIFI-16289
URL: https://issues.apache.org/jira/browse/NIFI-16289
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Mark Payne
Assignee: Mark Payne
{{PutCouchbaseIT.testPutDocument}} remains flaky after NIFI-16194.
Recent failure:
https://github.com/apache/nifi/actions/runs/33678350576/job/100452694901
The test creates a warmed Couchbase connection service, then
{{getTestDocument()}} opens a separate {{Cluster}} and immediately executes
{{collection.get()}}. The SDK can time out after 2.5 seconds with
{{BUCKET_OPEN_IN_PROGRESS}}.
Reuse the warmed service or wait for the bucket to be KV-ready before the
direct get. Review {{GetCouchbaseIT.insertTestDocument()}} for the same pattern.
h3. Problem
{{ClusteredConnectorBacklogIT#testClusteredReadQueueAttributesLoadBalancedConnection}}
fails intermittently in CI during test teardown, not in the test body itself:
{code}
org.apache.nifi.toolkit.client.NiFiClientException: Error deleting Connector:
Node localhost:5671 is unable to fulfill this request due to: Cannot delete
StandardConnectorNode[...name=BacklogReportingTestConnector, state=STOPPED]
because its Process Group has 82 FlowFiles queued; all FlowFiles must be
removed before it can be deleted.
{code}
Observed on CI run 33395377036 (ubuntu-24.04, Java 25), surfaced while
reviewing PR [#11599|https://github.com/apache/nifi/pull/11599]. The sibling
test
{{testClusteredReadQueueAttributesNormalConnection}} ({{DO_NOT_LOAD_BALANCE}})
has
not exhibited this failure - only the {{ROUND_ROBIN}} variant has, which points
toward Load Balancing as a factor.
h3. Suspected Root Cause
This is a timing-dependent failure that we have not been able to reproduce
on demand, so the following is our best working theory rather than a
confirmed diagnosis.
{{NiFiSystemIT#teardown()}} -> {{destroyFlow()}} ->
{{NiFiClientUtil#deleteConnectors()}}
calls {{purgeConnectorFlowFiles()}} (a drop request) immediately before deleting
each Connector. For a load-balanced connection
({{SocketLoadBalancedFlowFileQueue}}), FlowFiles can plausibly be "in flight"
between nodes: a background thread polls a FlowFile out of the local
{{RemoteQueuePartition}} to transmit it to the peer node, and the record does
not appear to be counted as present on either node's queue until the
transfer completes and the peer enqueues it. If a drop/purge request is
issued during that window, it could report completion without having
dropped those in-transit records, which would then land on the peer's queue
after the purge already ran - leaving FlowFiles queued when the connector
delete is subsequently attempted. This lines up with the symptom but has
not been directly observed happening.
{{ConnectionDTO#getLoadBalanceStatus()}} ({{LOAD_BALANCE_ACTIVE}} /
{{LOAD_BALANCE_INACTIVE}} / {{LOAD_BALANCE_NOT_CONFIGURED}}, backed by
{{FlowFileQueue#isActivelyLoadBalancing()}}) is the existing public signal for
this in-flight state.
h3. Investigation notes
Checked whether this could be related to NIFI-16262 (PR
[#11597|https://github.com/apache/nifi/pull/11597], "Fix
Connector stop handling for multiple callers"). That change only touches
start/stop race handling in {{StandardConnectorNode}} and does not touch
purge/drop-request handling, so it looks unrelated. PR
[#11599|https://github.com/apache/nifi/pull/11599]'s base
commit already includes the NIFI-16262 fix, and the failure still
reproduced there, which is consistent with the two being unrelated.
h3. Proposed Mitigation
Attempted to stabilize the test itself (no framework changes) in
{{ClusteredConnectorBacklogIT}}: wrapped the test body in try/finally and added
a private {{waitForLoadBalancingComplete()}} helper that polls each connection
in the Connector's flow via {{getConnectorClient().getFlow()}} until none report
{{LOAD_BALANCE_ACTIVE}}. This runs in the finally block, after
{{stopConnector()}},
with the intent that Load Balancing has settled before control returns to
teardown's purge - regardless of whether the test's own assertions passed.
This is a hypothesis-driven mitigation based on the suspected root cause
above, not a confirmed fix, since the original failure could not be
reproduced on demand to verify against directly. Kept local to this test
class rather than added as a general-purpose helper on {{NiFiSystemIT}}, since
the race is specific to this test's use of a load-balanced Connector queue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
