I found the issue, and I will share it here if anymore come here with the same problem :
The problem is permission, make sure you add securityContext: fsGroup: 65534 this is the user Prometheus uses behind the hood to write into files, see https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ for more details One thing that threw me off, is that Alertmanager does not attempt to persist the silences instantly, it first caches it and then some time later run the Maintenance, which writes it in a file. So the error log "Permission Denied" only appeared some 15 minutes after restarting my pod (that is why i missed this error log the first time). Le mercredi 2 décembre 2020 à 14:10:07 UTC+1, Flavio Deroo a écrit : > Hello all, > > We use Prometheus + Alertmanager inside an EKS cluster. The whole setup > works fine, except when we turn off the cluster at night, we lose the > silences. They are not persisted. > Prometheus and Alertmanager are deployed as StatefulSets, with a > volumeClaimTemplates. > > Prometheus data is persisted fine in EBS using a StorageClass item in > Kubernetes + volumeClaimTemplates in the StatefulSet + mountPath: > /prometheus > > But for alertmanager, it does not work. The EBS volume is mounted and > attached to the pod, but the data doesnt seem to get persisted. > > I have no clue why, and I have no idea how to debug this, no logs > > Version / image : prom/alertmanager:v0.20.0 > Arguments : > - --config.file=/etc/alertmanager/config.yml > - --storage.path=/alertmanager > > Mounted Volume : > volumeMounts: > - name: alertmanager > mountPath: /alertmanager > > Volume claim : > volumeClaimTemplates: > - metadata: > name: alertmanager > annotations: > volume.beta.kubernetes.io/storage-class: encrypted-alertmanager-ebs > spec: > accessModes: ["ReadWriteOnce"] > resources: > requests: > storage: 1Gi > > and finally storage class: > apiVersion: storage.k8s.io/v1 > kind: StorageClass > metadata: > name: encrypted-alertmanager-ebs > namespace: monitoring > provisioner: kubernetes.io/aws-ebs > > I understand this question touches both Alertmanager configuration and > Kubernetes, but I could not find a single definitive guide / doc on the > alertmanager storage persisting options, and therefore I think it can be > valuable for the community to solve this here > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/269e1c6f-8962-47cd-b4e8-5a65b0073c3bn%40googlegroups.com.
