To more easily debug your issue, please take Grafana out of the equation, as it has its own foibles. To do this, use the PromQL web browser to formulate a query within the PromQL web interface.
Then, show if there's a difference between the results: if there is, show the exact query you're giving to PromQL and the exact definition of the recording rule. Show both graphs, and highlight the differences. My *guess* is it's something to do with detected counter resets, i.e. example_metrics is not increasing monotonically. You can formulate queries to detect this. On Tuesday, 11 March 2025 at 13:48:54 UTC Kishore Kumar wrote: > Hi Prometheus users, > We are having a PromQL query and a Recording Rule that records > the PromQL, like the example given below. > > *-record: rest-server-recording-rule* > * expr: sum(increase(example_metric[1m])) by (kubernetes_container_name)* > > The Recording Rule Scrape interval and Evaluation interval are both 30 > seconds, and set in Prometheus configuration. > > We are seeing unexpected spikes in the data recorded by the recording > rule, whereas this *unexpected spike* is not present in the source > expression, like shown in the below graph (Used Grafana for comparison). > > Can we know the reason why this unexpected spike is being created by the > recording rule? We would like to know the explanation of how recording rule > captures the data of a query. > > Thanks for reading this message, have a great day. > > *Sum(increase) RawQuery* - data produced when we query the raw > *expression* directly. > *Recording Rules* - Data captured by the recording rule. > [image: image-2025-3-10_18-52-48.png] > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/0ff53a7d-3672-43a3-80aa-008d2b3821b2n%40googlegroups.com.
