On 09/02/2021 14:21, Thibaut Sarion wrote:
Hello everyone !
I've got a question regarding remote_write and federation.
I have two levels of prometheus : *level 1* with classic prometheus
instances and *level 2* with prometheus federating some metrics from
my *level 1*.
Both levels are remote_writing to my backend, but I need to find a way
not to write the same metrics twice in my backend.
From *level 1*, I remote_write every metrics to the backend, while
from the level 2, I need to only remote_write the new ones (calculated
on level 2) so I was planning to drop metrics on level 2 based on the
federate job name.
remote_write:
- url: "mybackend"
write_relabel_configs: # attempt to drop every federated metric
- source_labels: ['job_name']
regex: "federate"
action: drop
In most cases, this would work, but if a user writes this rule on
*level 2*
rules:
- record: new_metric_on_level_2
expr: sum without(random_label) (counter_from_level_1)
new_metric_on_level_2 will inherit the job=federate, thus being
dropped at remote_write but it shouldn't as it's a new metric on
*level 2 *and needs to be remote_write to my backend.
Note : I cannot modify the rule for
adding without(random_label,job) even if it solves the problem
(because I don't own the rules)
Do you have any suggestion on how I could solve this use case ? Thank
you so much for reading
Hi Thibault,
This puzzles me since I read your message.
In level 2, can't you immediately remove the label federate in job
definition? But the problem is still at level 2. So all recording rules
at level 2 should add a dedicated label "iwanttoremotewritethis". This
does not seem to be a feature. If you don't manage level 1 and 2, can't
you use some post-processing automation to inject labels "iamlevel1" /
"iamlevel2" in all recording rules?
Then if you have only label "iamlevel2", you know that you can
remote_write this metric.
--
Laurent CREPET.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bfdf2cdd-8183-9d1a-9b91-cd4555c689b1%40aurentc.xyz.
