Hi, in a big kubernetes cluster I have a very huge prometheus. Trying to
contain his memory requests and OOM, I noticed that in the kubelet metrics
there a very high cardinality label: "id". So added this configuration:
metric_relabel_configs:
- action: labeldrop
regex: "id"
I maneged like this to extremely reduce memory consuption, apparently at no
costs, all my dashboard continues to work.
However sending metrics via remote_write to Cortex lead to this:
Error on senting writerequest to Cortex: code 400 Body user=mia-platform:
series={__name__=\"container_cpu_system_seconds_total\", [...] },
timestamp=2021-05-12T09:42:06.89Z: out of order sample\n"
For this metric, totally removing id label lead to be not unique:
container_fs_writes_total{container="",device="/dev/sda",id="/system.slice/docker.service",image="",name="",namespace="",pod=""}
3.794105e+06 1620812171482
container_fs_writes_total{container="",device="/dev/sda",id="/system.slice/kubelet.service",image="",name="",namespace="",pod=""}
33558 1620812167445
Is there a method to contains this high cardinality lead Prometheus to use
an accettable amount of RAM?
Thanks
Thomas
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/d6100c46-d46f-4e67-8be8-7d01a9bd04cfn%40googlegroups.com.
