Hi everyone,
Today I have a problem about my rules expression because I try to join
metrics together to get the name of the swam node in it.
In order to do this, I have left_joined my metrics by node_id and get
node_name. It's works fine in the prometheus console but when I deploy my
rules, I get this message:
level=warn ts=2020-03-10T20:27:12.111Z caller=manager.go:525
component="rule manager" group="Container alert" msg="Evaluating rule
failed"
rule="alert: task_high_memory_usage_1g
expr: sum by(container_label_com_docker_swarm_task_name,
container_label_com_docker_swarm_node_id)
(container_memory_rss{container_label_com_docker_swarm_task_name=~".+"}) *
on(container_label_com_docker_swarm_node_id) group_left(node_name)
node_meta > 1e+06
err="found duplicate series for the match group
{container_label_com_docker_swarm_node_id=\"jilcpclonjg7jj1chh29b19pl\"} on
the right hand-side of the operation:
[{__name__=\"node_meta\",
container_label_com_docker_swarm_node_id=\"jilcpclonjg7jj1chh29b19pl\",
instance=\"10.0.9.10:9100\", job=\"node-exporter\",
node_id=\"jilcpclonjg7jj1chh29b19pl\", node_name=\"***\"},
{__name__=\"node_meta\",
container_label_com_docker_swarm_node_id=\"jilcpclonjg7jj1chh29b19pl\",
instance=\"10.0.8.20:9100\", job=\"node-exporter\",
node_id=\"jilcpclonjg7jj1chh29b19pl\", node_name=\"***\"}];
many-to-many matching not allowed: matching labels must be unique on one
side"
I have a label instance with two different values. my first metric is get
by cadvisor and the other is generated by node_exporter so two different
instances
node_meta{container_label_com_docker_swarm_node_id="jilcpclonjg7jj1chh29b19pl",
instance="10.0.9.10:9100", job="node-exporter",
node_id="jilcpclonjg7jj1chh29b19pl", node_name="***"}
This is an example of container_memory_rss:
container_memory_rss{container_label_caddy_version="1.0.3",container_label_com_docker_stack_namespace="supervision",container_label_com_docker_swarm_node_id="jilcpclonjg7jj1chh29b19pl",container_label_com_docker_swarm_service_id="5t7qdfsjdirecjvw0nlpzlw0w",container_label_com_docker_swarm_service_name="supervision_dockerd-exporter",container_label_com_docker_swarm_task_id="n8ji42apbj0t2bctr6ilgcvt2",container_label_com_docker_swarm_task_name="supervision_dockerd-exporter.jilcpclonjg7jj1chh29b19pl.n8ji42apbj0t2bctr6ilgcvt2",",id="/docker/312f83a134e144e3342d6229ec1a239e0539b77ec79f3a7e493f04fc9e24edf1",image="
registry.test.toto.fr/dockerd-exporter:1.0.3@sha256:31f18339414875c404647acac452883e2a7d6fdf89b8d8c8ae5e61140c392b80
",instance="10.0.9.20:8080
",job="cadvisor",name="supervision_dockerd-exporter.jilcpclonjg7jj1chh29b19pl.n8ji42apbj0t2bctr6ilgcvt2"}
The alert is sent anyway but I don't like have errors in app ^^
Maybe there is a way to delete instance label before doing the left_join, I
don't need this label in my expression.
Thanks for your time !
Best regards
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/b2e84601-b9f0-4343-bda7-fc8eafb80b4c%40googlegroups.com.
