> Yes, that's the problem; recommend configuring persistent storage for the
>Kubernetes containers running NiFi.
I have already configured Persistent Volume Claims (PVC's) for NiFi on
Kubernetes.
They work fine for persisting data between pod restarts.
The problem the limitation of PVCs on Kubernetes is that they are not
replicated between availability zones.
So if there is a permanent loss of the availability zone hosting the PVC then
it results in permanent data loss of the flow files.
My questions was about what options are available for backing up NiFi flow
files in Kubernetes?
I don't seem to be able to find a solution where NiFi flow files can be either
stored in PostgreSQL (which we already backup) or to S3-compatible object
storage.
I'm hoping someone with experience of running NiFi on Kubernetes may have the
answer.
Thanks.
On Monday, 23 October 2023 at 14:35:25 BST, Patrick Timmins
<[email protected]> wrote:
"The problem is that our PVCs aren't backed up, nor are they resilient to an
availability zone failure.
So I'm trying to find a method for backing up the flow files, but I've not
found anything for Kubernetes."
Yes, that's the problem; recommend configuring persistent storage for the
Kubernetes containers running NiFi.
On 10/23/2023 7:41 AM, David Dean via users wrote:
Hi everyone.
I'm running a single instance of NiFi on a Kubernetes cluster using the the
Helm chart from Cetic (I can't share the link here, it's on GitHub).
This Helm chart uses PVCs for flow file storage, which works fine for
persisting flows between restarts of NiFi.
The problem is that our PVCs aren't backed up, nor are they resilient to an
availability zone failure.
So I'm trying to find a method for backing up the flow files, but I've not
found anything for Kubernetes.
Ideally I would like to write the flow files to a PostgreSQL database, which
is already present in the Kubernetes cluster and is backed up. But from what I
can read (and I couldn't easily be wrong!) I don't think NiFi can persist flow
files in a PostgreSQL database?
The other option would be to write the flow files to object storage, like an
S3 or Azure blob storage bucket, but I don't think NiFi can do that either?
My question for anyone else running NiFi on Kubernetes - how do you back up
the flow files so they can be resolved in the event that the PVCs are
permanently lost?
Thanks.
