Hello all, We use Prometheus + Alertmanager inside an EKS cluster. The whole setup works fine, except when we turn off the cluster at night, we lose the silences. They are not persisted. Prometheus and Alertmanager are deployed as StatefulSets, with a volumeClaimTemplates.
Prometheus data is persisted fine in EBS using a StorageClass item in Kubernetes + volumeClaimTemplates in the StatefulSet + mountPath: /prometheus But for alertmanager, it does not work. The EBS volume is mounted and attached to the pod, but the data doesnt seem to get persisted. I have no clue why, and I have no idea how to debug this, no logs Version / image : prom/alertmanager:v0.20.0 Arguments : - --config.file=/etc/alertmanager/config.yml - --storage.path=/alertmanager Mounted Volume : volumeMounts: - name: alertmanager mountPath: /alertmanager Volume claim : volumeClaimTemplates: - metadata: name: alertmanager annotations: volume.beta.kubernetes.io/storage-class: encrypted-alertmanager-ebs spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi and finally storage class: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: encrypted-alertmanager-ebs namespace: monitoring provisioner: kubernetes.io/aws-ebs I understand this question touches both Alertmanager configuration and Kubernetes, but I could not find a single definitive guide / doc on the alertmanager storage persisting options, and therefore I think it can be valuable for the community to solve this here -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6b4a1273-fd2d-47a9-b0b7-06a347b838ben%40googlegroups.com.
