Hello I fix my issue, For the record, I add retentionSize limits to 45 GB (my persistent volume use 50GB)
After that, prometheus starts to work normally and I think the root cause of my issue was one corrupt chunk of data level=error caller=head.go:785 component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: corruption in head chunk file /prometheus/chunks_head/000380: head chunk file has some unread data, but doesn't include enough bytes to read the chunk header - required:67104769, available:67104768, file:380" level=info component=tsdb msg="Deleting mmapped chunk files" level=info 5 component=tsdb msg="Deletion of mmap chunk files successful, reattempting m-mapping the on-disk chunks" level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=6.63822568s level=info component=tsdb msg="Replaying WAL, this may take a while El lunes, 6 de septiembre de 2021 a las 14:26:37 UTC-6, David B G. escribió: > Hello everybody, nice to meet you all > > I have a quick question regarding Prometheus using the kube prometheus > stack deployment <https://github.com/prometheus-operator/kube-prometheus>. > > and recently have an issue regarding the retention of metrics, my issue is > like this post <https://github.com/prometheus/prometheus/issues/8005> > that its appear my retention of time series exceeds the limit of 10 days > and its causing a disk usage of 100%. > > After making an exec inside the pod it seems that there are blocks that > have more than 10 days of retention. > > Aug 24 01FDVX28YSKFRCMDCRP4111VBJ > > Aug 24 11:03 01FDVX4XK0ACRPPZ65KM2YFJRZ > > Aug 24 13:00 01FDW3Y06JWQ81DG1XF59JR944 > > > > So, I have the next naive question: > > Is there any configuration I am missing? > > It’s possible that after the pod of Prometheus restart its counts again 10 > days from the time the pod gets reset? > > Its safe to delete manually my old blocks with a command like rm > block_name? > > I am using the next configuration of kube prometheus: > > Metric retention > > retention: 10d > > retentionSize: "" > > walCompression: false > > > > I am using Prometheus version 2.28.1 > > > > Any advise or help I will appreciate it 😊 > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/06afc7ca-37b7-4fc0-960f-2e2a5a7d2d2an%40googlegroups.com.
