[
https://issues.apache.org/jira/browse/NIFI-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Joseph updated NIFI-16593:
-------------------------------
Description:
We have nifi deployed using Kubernetes clustering in our product. We're on the
version 2.12
We use Wait and Notify many of our flow definitions. we have noticed that
sometimes when one of our main flow definition is deployed
Flow files are stuck in the NiFi wait queue. This was causing the HTTP calls to
return HTTP 504. NiFi does not recover until the queue is cleared.
Upon pod investigation we found that requests going on POD1 is not available in
POD2
Attaching logs
Example
In Nifi pod 0 , this id "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}" shows
"wait expired" at 2026-09-02T06:13:10.293Z
{code:java}
$ kubectl logs nifi-0 -n ns|grep "ddda8d71-ceea-4e17-815e-381dbe2b0e30"
Defaulted container "nifi-server"
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-0", "time":"2026-09-02T06:13:10.293Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-1] o.a.nifi.processors.standard.LogMessage
LogMessage[id=7203d32c-01cc-33b8-b6e9-1c9e25057522]
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Wait Expired"}
{code}
In Nifi Pod 1, this id "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}" shows
Completed and Notified at around 2026-09-02T02:03:20.440Z
{code:java}
kubectl logs nifi-1 -n ns|grep "ddda8d71-ceea-4e17-815e-381dbe2b0e30"
Defaulted container "nifi-server"
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:12.843Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: Acknowledged]"}
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:12.901Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: InProgress]"}
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:20.307Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-4] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: Completed]"}
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:20.440Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage
LogMessage[id=b450a79d-4747-32b6-a5cc-f52d74923bde]
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Notified"}
{code}
was:
We have nifi deployed using Kubernetes clustering in our product. We're on the
version 2.12
We use Wait and Notify many of our flow definitions. we have noticed that
sometimes when one of our main flow definition is deployed
Flow files are stuck in the NiFi wait queue. This was causing the HTTP calls to
return HTTP 504. NiFi does not recover until the queue is cleared.
Upon pod investigation we found that requests going on POD1 is not available in
POD2
attaching logs
Example
In Nifi pod 0 , this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows
"wait expired" at 2026-09-02T06:13:10.293Z
$ kubectl logs nifi-0 -n ns|grep "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"
{code:java}
Defaulted container "nifi-server"
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-0", "time":"2026-09-02T06:13:10.293Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-1] o.a.nifi.processors.standard.LogMessage
LogMessage[id=7203d32c-01cc-33b8-b6e9-1c9e25057522]
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Wait Expired"}
{code}
In Nifi Pod 1, this order id "ddda8d71-ceea-4e17-815e-381dbe2b0e30" shows
Completed and Notified at around 2026-09-02T02:03:20.440Z
kubectl logs nifi-1 -n ns|grep "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}"
{code:java}
Defaulted container "nifi-server"
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:12.843Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: Acknowledged]"}
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:12.901Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-2] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: InProgress]"}
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:20.307Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-4] o.a.nifi.processors.standard.LogMessage
LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
orderState: Completed]"}
{"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
"systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
"system":"nifi-1", "time":"2026-09-02T02:03:20.440Z" ,"timezone":"UTC",
"log":"[Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage
LogMessage[id=b450a79d-4747-32b6-a5cc-f52d74923bde]
SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Notified"}
{code}
> Wait and Notify sync issue for NiFi when deployed in cluster mode using
> Kubernetes clustering
> ---------------------------------------------------------------------------------------------
>
> Key: NIFI-16593
> URL: https://issues.apache.org/jira/browse/NIFI-16593
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.10.0, 2.12.0
> Reporter: John Joseph
> Priority: Major
>
> We have nifi deployed using Kubernetes clustering in our product. We're on
> the version 2.12
> We use Wait and Notify many of our flow definitions. we have noticed that
> sometimes when one of our main flow definition is deployed
>
> Flow files are stuck in the NiFi wait queue. This was causing the HTTP calls
> to return HTTP 504. NiFi does not recover until the queue is cleared.
>
> Upon pod investigation we found that requests going on POD1 is not available
> in POD2
>
> Attaching logs
>
> Example
> In Nifi pod 0 , this id "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}" shows
> "wait expired" at 2026-09-02T06:13:10.293Z
>
>
> {code:java}
> $ kubectl logs nifi-0 -n ns|grep "ddda8d71-ceea-4e17-815e-381dbe2b0e30"
> Defaulted container "nifi-server"
> {"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
> "system":"nifi-0", "time":"2026-09-02T06:13:10.293Z" ,"timezone":"UTC",
> "log":"[Timer-Driven Process Thread-1]
> o.a.nifi.processors.standard.LogMessage
> LogMessage[id=7203d32c-01cc-33b8-b6e9-1c9e25057522]
> SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Wait Expired"}
> {code}
>
>
> In Nifi Pod 1, this id "{*}ddda8d71-ceea-4e17-815e-381dbe2b0e30{*}" shows
> Completed and Notified at around 2026-09-02T02:03:20.440Z
>
> {code:java}
> kubectl logs nifi-1 -n ns|grep "ddda8d71-ceea-4e17-815e-381dbe2b0e30"
> Defaulted container "nifi-server"
> {"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
> "system":"nifi-1", "time":"2026-09-02T02:03:12.843Z" ,"timezone":"UTC",
> "log":"[Timer-Driven Process Thread-2]
> o.a.nifi.processors.standard.LogMessage
> LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
> notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
> orderState: Acknowledged]"}
> {"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
> "system":"nifi-1", "time":"2026-09-02T02:03:12.901Z" ,"timezone":"UTC",
> "log":"[Timer-Driven Process Thread-2]
> o.a.nifi.processors.standard.LogMessage
> LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
> notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
> orderState: InProgress]"}
> {"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
> "system":"nifi-1", "time":"2026-09-02T02:03:20.307Z" ,"timezone":"UTC",
> "log":"[Timer-Driven Process Thread-4]
> o.a.nifi.processors.standard.LogMessage
> LogMessage[id=a1aa0253-219a-3552-b745-47860eaed524] [ CMPLT] Completion
> notification is received [orderId: ddda8d71-ceea-4e17-815e-381dbe2b0e30,
> orderState: Completed]"}
> {"type":"log", "host":"labname", "level":"WARN", "event-type":"N_USER_OPER",
> "systemid":"norcnifi","neid":"c6783854f1644d8080aa47852268d7d1",
> "system":"nifi-1", "time":"2026-09-02T02:03:20.440Z" ,"timezone":"UTC",
> "log":"[Timer-Driven Process Thread-3]
> o.a.nifi.processors.standard.LogMessage
> LogMessage[id=b450a79d-4747-32b6-a5cc-f52d74923bde]
> SQ->ddda8d71-ceea-4e17-815e-381dbe2b0e30->Notified"}
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)