[ 
https://issues.apache.org/jira/browse/NIFI-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Joseph updated NIFI-16593:
-------------------------------
    Description: 
We have nifi deployed using Kubernetes clustering in our product. We're on the 
version 2.12

We use Wait and Notify many of our flow definitions. we have noticed that 
sometimes when one of our main flow definition is deployed 

 

Flow files are stuck in the NiFi wait queue. This was causing the HTTP calls to 
return HTTP 504. NiFi does not recover until the queue is cleared.

 

Upon pod investigation we found that requests going on POD1 is not available in 
POD2 

 

attaching logs

 

Example
In Nifi pod 0 , this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows 
"wait expired" at 2026-09-02T06:13:10.293Z

$ kubectl logs nifi-0 -n ns|grep "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"

 
{code:java}
Defaulted container "nifi-server"
{"type":"log", "host":"ao0031node09.ao0031", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-0", "time":"2026-09-02T06:13:10.293Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-1] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=7203d32c-01cc-33b8-b6e9-1c9e25057522] 
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Wait Expired"} 
{code}
 

 

In Nifi Pod 1, this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows 
Completed and Notified at around 2026-09-02T02:03:20.440Z

 kubectl logs nifi-1 -n ns|grep "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"

 

 
{code:java}
Defaulted container "nifi-server" 
{"type":"log", "host":"ao0031node12.ao0031", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:12.843Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: Acknowledged]"}
{"type":"log", "host":"ao0031node12.ao0031", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:12.901Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: InProgress]"}
{"type":"log", "host":"ao0031node12.ao0031", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:20.307Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-4] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: Completed]"}
{"type":"log", "host":"ao0031node12.ao0031", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:20.440Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=b450a79d-4747-32b6-a5cc-f52d74923bde] 
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Notified"}
{code}
 

 

 

 

 

 

  was:
We have nifi deployed using Kubernetes clustering in our product. We're on the 
version 2.12

We use Wait and Notify many of our flow definitions. we have noticed that 
sometimes when one of our main flow definition is deployed 

 

Flow files are stuck in the NiFi wait queue. This was causing the HTTP calls to 
return HTTP 504. NiFi does not recover until the queue is cleared.

 

Upon pod investigation we found that requests going on POD1 is not available in 
POD2 

 

attaching logs

 

Example
In Nifi pod 0 , this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows 
"wait expired" at 2026-09-02T06:13:10.293Z

$ kubectl logs nifi-0 -n doc-norc|grep 
"{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"

 
{code:java}
Defaulted container "nifi-server" out of: nifi-server, nifi-metrics, 
cbura-sidecar
{"type":"log", "host":"ao0031node09.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-0", "time":"2026-09-02T06:13:10.293Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-1] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=7203d32c-01cc-33b8-b6e9-1c9e25057522] 
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Wait Expired"} 
{code}
 

 


In Nifi Pod 1, this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows 
Completed and Notified at around 2026-09-02T02:03:20.440Z

 kubectl logs nifi-1 -n doc-norc|grep 
"{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"

 

 
{code:java}
Defaulted container "nifi-server" out of: nifi-server, nifi-metrics, 
cbura-sidecar
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:12.843Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: Acknowledged]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:12.901Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: InProgress]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:20.307Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-4] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
orderState: Completed]"}
{"type":"log", "host":"ao0031node12.ao0031.tre.nsn-rdnet.net", "level":"WARN", 
"event-type":"N_USER_OPER", 
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
"system":"nifi-1", "time":"2026-09-02T02:03:20.440Z" ,"timezone":"UTC", 
"log":"[Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage 
LogMessage[id=b450a79d-4747-32b6-a5cc-f52d74923bde] 
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Notified"}
{code}
 

 

 

 

 

 


> Wait and Notify sync issue for NiFi when deployed in cluster mode using 
> Kubernetes clustering
> ---------------------------------------------------------------------------------------------
>
>                 Key: NIFI-16593
>                 URL: https://issues.apache.org/jira/browse/NIFI-16593
>             Project: Apache NiFi
>          Issue Type: Bug
>    Affects Versions: 2.10.0, 2.12.0
>            Reporter: John Joseph
>            Priority: Major
>
> We have nifi deployed using Kubernetes clustering in our product. We're on 
> the version 2.12
> We use Wait and Notify many of our flow definitions. we have noticed that 
> sometimes when one of our main flow definition is deployed 
>  
> Flow files are stuck in the NiFi wait queue. This was causing the HTTP calls 
> to return HTTP 504. NiFi does not recover until the queue is cleared.
>  
> Upon pod investigation we found that requests going on POD1 is not available 
> in POD2 
>  
> attaching logs
>  
> Example
> In Nifi pod 0 , this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows 
> "wait expired" at 2026-09-02T06:13:10.293Z
> $ kubectl logs nifi-0 -n ns|grep "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"
>  
> {code:java}
> Defaulted container "nifi-server"
> {"type":"log", "host":"ao0031node09.ao0031", "level":"WARN", 
> "event-type":"N_USER_OPER", 
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
> "system":"nifi-0", "time":"2026-09-02T06:13:10.293Z" ,"timezone":"UTC", 
> "log":"[Timer-Driven Process Thread-1] 
> o.a.nifi.processors.standard.LogMessage 
> LogMessage[id=7203d32c-01cc-33b8-b6e9-1c9e25057522] 
> SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Wait Expired"} 
> {code}
>  
>  
> In Nifi Pod 1, this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows 
> Completed and Notified at around 2026-09-02T02:03:20.440Z
>  kubectl logs nifi-1 -n ns|grep "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"
>  
>  
> {code:java}
> Defaulted container "nifi-server" 
> {"type":"log", "host":"ao0031node12.ao0031", "level":"WARN", 
> "event-type":"N_USER_OPER", 
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
> "system":"nifi-1", "time":"2026-09-02T02:03:12.843Z" ,"timezone":"UTC", 
> "log":"[Timer-Driven Process Thread-2] 
> o.a.nifi.processors.standard.LogMessage 
> LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
> notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
> orderState: Acknowledged]"}
> {"type":"log", "host":"ao0031node12.ao0031", "level":"WARN", 
> "event-type":"N_USER_OPER", 
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
> "system":"nifi-1", "time":"2026-09-02T02:03:12.901Z" ,"timezone":"UTC", 
> "log":"[Timer-Driven Process Thread-2] 
> o.a.nifi.processors.standard.LogMessage 
> LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
> notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
> orderState: InProgress]"}
> {"type":"log", "host":"ao0031node12.ao0031", "level":"WARN", 
> "event-type":"N_USER_OPER", 
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
> "system":"nifi-1", "time":"2026-09-02T02:03:20.307Z" ,"timezone":"UTC", 
> "log":"[Timer-Driven Process Thread-4] 
> o.a.nifi.processors.standard.LogMessage 
> LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion 
> notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30, 
> orderState: Completed]"}
> {"type":"log", "host":"ao0031node12.ao0031", "level":"WARN", 
> "event-type":"N_USER_OPER", 
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1", 
> "system":"nifi-1", "time":"2026-09-02T02:03:20.440Z" ,"timezone":"UTC", 
> "log":"[Timer-Driven Process Thread-3] 
> o.a.nifi.processors.standard.LogMessage 
> LogMessage[id=b450a79d-4747-32b6-a5cc-f52d74923bde] 
> SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Notified"}
> {code}
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to