John Joseph created NIFI-16593:
----------------------------------
Summary: Wait and Notify sync issue for NiFi when deployed in
cluster mode using Kubernetes clustering
Key: NIFI-16593
URL: https://issues.apache.org/jira/browse/NIFI-16593
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.12.0, 2.10.0
Reporter: John Joseph
We have nifi deployed using Kubernetes clustering in our product. We're on the
version 2.12
We use Wait and Notify many of our flow definitions. we have noticed that
sometimes when one of our main flow definition is deployed
Flow files are stuck in the NiFi wait queue. This was causing the HTTP calls to
return HTTP 504. NiFi does not recover until the queue is cleared.
Upon pod investigation we found that requests going on POD1 is not available in
POD2
attaching logs
Example
In Nifi pod 0 , this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows
"wait expired" at 2026-09-02T06:13:10.293Z
$ kubectl logs nifi-0 -n doc-norc|grep
"{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"
{code:java}
Defaulted container "nifi-server" out of: nifi-server, nifi-metrics,
cbura-sidecar
{"type":"log", "host":"ao0031node09.ao0031.tre.nsn-rdnet.net", "level":"WARN",
"event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-0", "time":"2026-09-02T06:13:10.293Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-1] o.a.nifi.processors.standard.LogMessage
LogMessage[id=7203d32c-01cc-33b8-b6e9-1c9e25057522]
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Wait Expired"}
{code}
In Nifi Pod 1, this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows
Completed and Notified at around 2026-09-02T02:03:20.440Z
kubectl logs nifi-1 -n doc-norc|grep
"{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"
{code:java}
Defaulted container "nifi-server" out of: nifi-server, nifi-metrics,
cbura-sidecar
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN",
"event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:12.843Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: Acknowledged]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN",
"event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:12.901Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: InProgress]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN",
"event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:20.307Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-4] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: Completed]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN",
"event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:20.440Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage
LogMessage[id=b450a79d-4747-32b6-a5cc-f52d74923bde]
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Notified"}
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)