Hi all,
In Grafana I have something like:
avg(
label_replace(n_http_request_sec {n_instance="$instance"} offset 1d,
"offset", "1d", "__name__", ".*") or
label_replace(n_http_request_sec {n_instance="$instance"} offset 2d,
"offset", "2d", "__name__", ".*") or
label_replace(n_http_request_sec {n_instance="$instance"} offset 3d,
"offset", "3d", "__name__", ".*") or
label_replace(n_http_request_sec {n_instance="$instance"} offset 4d,
"offset", "4d", "__name__", ".*") or
label_replace(n_http_request_sec {n_instance="$instance"} offset 5d,
"offset", "5d", "__name__", ".*") or
label_replace(n_http_request_sec {n_instance="$instance"} offset 6d,
"offset", "6d", "__name__", ".*") or
label_replace(n_http_request_sec {n_instance="$instance"} offset 7d,
"offset", "7d", "__name__", ".*")
) without (offset, instance, job)
to compute some anomaly detection (more info on
https://groups.google.com/d/msgid/prometheus-users/fe876c1e-ded7-45e4-974e-ddb0b916a675%40googlegroups.com)
and it works for my workload, so I am moving that to a recorded prometheus
metrics with:
# Avg over last week's samples this time
- record: job:nginx_http_req_sec:avg_1w
expr: avg(
label_replace(n_http_request_sec offset 1d, "offset", "1d",
"__name__", ".*") or
label_replace(n_http_request_sec offset 2d, "offset", "2d",
"__name__", ".*") or
label_replace(n_http_request_sec offset 3d, "offset", "3d",
"__name__", ".*") or
label_replace(n_http_request_sec offset 4d, "offset", "4d",
"__name__", ".*") or
label_replace(n_http_request_sec offset 5d, "offset", "5d",
"__name__", ".*") or
label_replace(n_http_request_sec offset 6d, "offset", "6d",
"__name__", ".*") or
label_replace(n_http_request_sec offset 7d, "offset", "7d",
"__name__", ".*")
) without (offset, instance, job)
What I see is that there is a "delay" in the data between the grafana panel
and the datapoints generated from job:nginx_http_req_sec:avg_1w,
where the datapoints of job:nginx_http_req_sec:avg_1w are the same as the
grafana ones shifted forward of 30s.
Is this something related to the scrape_interval or to some other interval
variable?
Relevant prometheus config:
global:
scrape_interval: 30s
scrape_timeout: 15s
evaluation_interval: 30s
Edit: C&P and format.
Thanks in advance,
d.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/d8c976fd-acfe-4c74-8427-4688b29a8d89%40googlegroups.com.
