As short term solution I'm recording something like:
label_replace(n_http_request_sec offset 259170s, "offset", "259170s",
"__name__", ".*")
instead of:
label_replace(n_http_request_sec offset 3d, "offset", "3d", "__name__",
".*")
where 259170s = 3d - 30s (evaluation_interval)
and it seems working.
d.
On Monday, 18 May 2020 08:34:36 UTC+1, Davide D'Amico wrote:
>
> Hi all,
>
> In Grafana I have something like:
>
> avg(
>
> label_replace(n_http_request_sec {n_instance="$instance"} offset 1d,
> "offset", "1d", "__name__", ".*") or
>
> label_replace(n_http_request_sec {n_instance="$instance"} offset 2d,
> "offset", "2d", "__name__", ".*") or
>
> label_replace(n_http_request_sec {n_instance="$instance"} offset 3d,
> "offset", "3d", "__name__", ".*") or
>
> label_replace(n_http_request_sec {n_instance="$instance"} offset 4d,
> "offset", "4d", "__name__", ".*") or
>
> label_replace(n_http_request_sec {n_instance="$instance"} offset 5d,
> "offset", "5d", "__name__", ".*") or
>
> label_replace(n_http_request_sec {n_instance="$instance"} offset 6d,
> "offset", "6d", "__name__", ".*") or
>
> label_replace(n_http_request_sec {n_instance="$instance"} offset 7d,
> "offset", "7d", "__name__", ".*")
>
> ) without (offset, instance, job)
>
> to compute some anomaly detection (more info on
> https://groups.google.com/d/msgid/prometheus-users/fe876c1e-ded7-45e4-974e-ddb0b916a675%40googlegroups.com)
>
> and it works for my workload, so I am moving that to a recorded prometheus
> metrics with:
>
> # Avg over last week's samples this time
>
> - record: job:nginx_http_req_sec:avg_1w
>
> expr: avg(
>
> label_replace(n_http_request_sec offset 1d, "offset",
> "1d", "__name__", ".*") or
>
> label_replace(n_http_request_sec offset 2d, "offset",
> "2d", "__name__", ".*") or
>
> label_replace(n_http_request_sec offset 3d, "offset",
> "3d", "__name__", ".*") or
>
> label_replace(n_http_request_sec offset 4d, "offset",
> "4d", "__name__", ".*") or
>
> label_replace(n_http_request_sec offset 5d, "offset",
> "5d", "__name__", ".*") or
>
> label_replace(n_http_request_sec offset 6d, "offset",
> "6d", "__name__", ".*") or
>
> label_replace(n_http_request_sec offset 7d, "offset",
> "7d", "__name__", ".*")
>
> ) without (offset, instance, job)
>
>
>
> What I see is that there is a "delay" in the data between the grafana
> panel and the datapoints generated from job:nginx_http_req_sec:avg_1w,
>
> where the datapoints of job:nginx_http_req_sec:avg_1w are the same as the
> grafana ones shifted forward of 30s.
>
>
>
> Is this something related to the scrape_interval or to some other interval
> variable?
>
>
>
> Relevant prometheus config:
>
> global:
>
> scrape_interval: 30s
>
> scrape_timeout: 15s
>
> evaluation_interval: 30s
>
>
>
> Edit: C&P and format.
>
> Thanks in advance,
>
> d.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/08f829a4-fefd-47af-8b9a-df991aaaf146%40googlegroups.com.
