Re: [prometheus-developers] How. to deal with outdated series in alerts.
Thank you so much. It helped me. On Monday, November 16, 2020 at 1:23:08 PM UTC+5:30 matt...@prometheus.io wrote: > Restart the application that produces metrics. > > Generally, the client libraries will remember metrics even if they are not > being incremented anymore (it cannot know that they won't be again). > Restarting clears the "seen" label set in each process. > > Side note: this can also bite you the other way – if an endpoint is > unexpectedly never being requested after a restart (say, it wasn't hooked > up correctly in the code), this alert will not detect it, because generic > instrumentation of HTTP calls cannot know which valid paths there *should* > be. > > In some cases, we alert like this because we want to check if a specific > business action has occurred. Instead of relying on the automatic HTTP > metrics, we can separately instrument the code with another metric that we > control, and where we can "increment by zero" for all possible label > combinations on startup. > > /MR > > On Sun, Nov 15, 2020, 17:00 Mounya A wrote: > >> Hello all, >>I have a question - how do we deal with labels that are no longer >> there (intentionally stopped) in alerts. Will there be any threshold time >> to consider it as stale or unwanted. >>I have configured an alert when rate(requests[1m]) == 0. It is >> firing alerts for labels, that didn't show up in the past 7 days. I have >> intentionally stopped and don't want to alert in such conditions. How to >> deal with this. >> >> Thanks in advance. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to prometheus-devel...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/d7c7ed77-0f69-4a5a-b36a-40c95c669546n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-developers/d7c7ed77-0f69-4a5a-b36a-40c95c669546n%40googlegroups.com?utm_medium=email_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/ccf7f805-c625-4206-823c-9dcf250a9221n%40googlegroups.com.
[prometheus-developers] How. to deal with outdated series in alerts.
Hello all, I have a question - how do we deal with labels that are no longer there (intentionally stopped) in alerts. Will there be any threshold time to consider it as stale or unwanted. I have configured an alert when rate(requests[1m]) == 0. It is firing alerts for labels, that didn't show up in the past 7 days. I have intentionally stopped and don't want to alert in such conditions. How to deal with this. Thanks in advance. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d7c7ed77-0f69-4a5a-b36a-40c95c669546n%40googlegroups.com.
[prometheus-developers] Calculating quantiles with summaries.
Hello all,
I have a question regarding calculting qunatiles in prometheus. I
can only see every body mentioning the use of histograms to calculate
qunatiles .We have quantile function in promql right? . Is it appropriate
to use this function with summaries?
Query I am using.
quantile(.99 ,
(sum by(job)(rate(api_response_time_sum{job="$stack" ,
service=~"$product"}[30d])) /
sum by(job)(rate(api_response_time_count{job="$stack",
service=~"$product"}[30d]))
)
)
will the above query give the desired result?
Thanks in Advance .
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/56763331-fa1a-4fd3-b7a1-07d2eaaf53d2n%40googlegroups.com.
Re: [prometheus-developers] Multiprocessing of python_client for independent python deamons
Thanks for the reply Brian. As far as I know, most of the disadvantages are caused by lot of process churn, ours are 3-5 long lived processes and we need only simple metrics like Counter and Summary. Still ,will we be affected by those disadvantages? Are there any limitations other than those mentioned in the documentation??Thanks in advance. On Friday, August 28, 2020 at 12:14:30 PM UTC+5:30 brian@robustperception.io wrote: > On Fri, 28 Aug 2020 at 07:39, Mounya A wrote: > >> Hii everyone, >> We have multiple independent deamons of same application in a >> machine. I would like to serve aggregated metrics from a single port , >> rather than starting server in each deamon and treating them as a separate >> time series. I can handle the exposition part with try.. except so that the >> first deamon that gets hold of the port can start the server. My doubt is >> regarding the correctness of metrics data if i use the python client's >> multiprocessing support. (as it is not a direct use of gunicorn or >> multiprocessing module) . Is there any chance of ending up with incorrect >> data if i use this approach . Thanks in advance. > > > If they're independent daemons then each should have their own /metrics. > The multiprocessing mode comes with a number of disadvantages, and should > only be used if you've no other option. > > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b1724855-ac9d-4f47-a7d4-35440eeddae5n%40googlegroups.com.
[prometheus-developers] Multiprocessing of python_client for independent python deamons
Hii everyone, We have multiple independent deamons of same application in a machine. I would like to serve aggregated metrics from a single port , rather than starting server in each deamon and treating them as a separate time series. I can handle the exposition part with try.. except so that the first deamon that gets hold of the port can start the server. My doubt is regarding the correctness of metrics data if i use the python client's multiprocessing support. (as it is not a direct use of gunicorn or multiprocessing module) . Is there any chance of ending up with incorrect data if i use this approach . Thanks in advance. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/c114f597-e1cc-4ca8-81d4-fbb2f15c3464n%40googlegroups.com.
