Hi Joe, I agree the system can recover after the restart, but we want to deliver data with as little latency as possible, so if FlowFiles stay queued during the restart time window, by the time the node is up and running again the data will be very late when time it finally leaves the NiFi cluster.
Regards, Alexis. On Sat, 17 Aug 2024, 16:59 Joe Witt, <[email protected]> wrote: > Alexis > > > If you are simply restarting nodes you dont need to ensure everything is > processed first. It will recover. > > If you want to scale down by removing a node there is an api to invoke > offloading so you could scale down to a single node even if needed. > > Thanks > > On Sat, Aug 17, 2024 at 2:02 AM Alexis Sarda-Espinosa < > [email protected]> wrote: > >> Hello, >> >> We use NiFi in cluster mode in Kubernetes via a StatefulSet. Sometimes, >> we need to restart the cluster, so the StatefulSet restarts one pod at a >> time. In this scenario, will a given node attempt to process all >> "in-flight" FlowFiles to drain the queues as much as possible before >> shutting down? >> >> I know it's probably impossible to have a completely graceful offload in >> these cases, but since the process still receives a termination signal and >> has some time before it must stop, it could try to prevent FlowFiles from >> staying in the system for too long, but I don't know if there are any >> mechanisms to handle this in NiFi. >> >> Regards, >> Alexis. >> >
