John Joseph created NIFI-16593:
----------------------------------

             Summary: Wait and Notify sync issue for NiFi when deployed in 
cluster mode using Kubernetes clustering
                 Key: NIFI-16593
                 URL: https://issues.apache.org/jira/browse/NIFI-16593
             Project: Apache NiFi
          Issue Type: Bug
    Affects Versions: 2.12.0, 2.10.0
            Reporter: John Joseph


We have nifi deployed using Kubernetes clustering in our product. We're on the 
version 2.12

We use Wait and Notify many of our flow definitions. we have noticed that 
sometimes when one of our main flow definition is deployed 

 

Flow files are stuck in the NiFi wait queue. This was causing the HTTP calls to 
return HTTP 504. NiFi does not recover until the queue is cleared.

 

Upon pod investigation we found that requests going on POD1 is not available in 
POD2 

 

attaching logs

 

Example
In Nifi pod 0 , this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows 
"wait expired" at 2026-09-02T06:13:10.293Z

$ kubectl logs nifi-0 -n doc-norc|grep 
"{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"

 
{code:java}
Defaulted container "nifi-server" out of: nifi-server, nifi-metrics, 
cbura-sidecar
{"type":"log", "host":"ao0031node09.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-0", "time":"2026-09-02T06:13:10.293Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-1] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=7203d32c-01cc-33b8-b6e9-1c9e25057522] 
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Wait Expired"} 
{code}
 

 


In Nifi Pod 1, this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows 
Completed and Notified at around 2026-09-02T02:03:20.440Z

 kubectl logs nifi-1 -n doc-norc|grep 
"{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"

 

 
{code:java}
Defaulted container "nifi-server" out of: nifi-server, nifi-metrics, 
cbura-sidecar
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:12.843Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: Acknowledged]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:12.901Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: InProgress]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:20.307Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-4] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: Completed]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:20.440Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=b450a79d-4747-32b6-a5cc-f52d74923bde] 
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Notified"}
{code}
 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to