Re: [prometheus-users] Metrics vs log level

On Sat, Oct 8, 2022 at 4:22 AM Muthuveerappan Periyakaruppan < muthu.veerap...@gmail.com> wrote: > Please find replies inline. > > On Friday, 7 October, 2022 at 1:25:27 pm UTC+5:30 Stuart Clark wrote: > >> On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote: >> > we have a situation , where w

Re: [prometheus-users] Metrics vs log level

So, different opinion here. Metrics are meant to tell you _when_ to debug. They're not meant to always be debugging itself. Metrics are supposed to tell you when it's time to get out other tools. * Look at the logs * Look at traces * Look at profilers Trying to get every dimension on every metri

Re: [prometheus-users] Metrics vs log level

Thanks a lot Fabian, will check out and get back to you. On Friday, 7 October, 2022 at 3:35:28 pm UTC+5:30 fab...@fstab.de wrote: > Hi, > > Funny, I commented the same idea of having two separate registries on > https://github.com/prometheus/client_java/issues/815 this morning. > > Currently y

Re: [prometheus-users] Metrics vs log level

Interesting CollectorRegistryCollector On Friday, 7 October, 2022 at 3:13:50 pm UTC+5:30 matt...@prometheus.io wrote: > > Say, If we write a wrapper on top of prometheus java client API, its > going to be messy > > You can make it relatively clean by creating (and incrementing) all the > metr

Re: [prometheus-users] Metrics vs log level

We already dropped few that are unused ... even after that we ended up in that number ... Is there a way to find out / query unused metrics or least queried ones ? topk somehow does not work for us ... it gives different error every time. not able to spend time on that. do you have , any handy

Re: [prometheus-users] Metrics vs log level

Please find replies inline. On Friday, 7 October, 2022 at 1:25:27 pm UTC+5:30 Stuart Clark wrote: > On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote: > > we have a situation , where we have 8 to 15 million head series in > > each Prometheus and we have 7 instance of them (federated). O

Re: [prometheus-users] Metrics vs log level

Hi, Funny, I commented the same idea of having two separate registries on https://github.com/prometheus/client_java/issues/815 this morning. Currently you would just register each metric with two registries manually errorLevelRegistry.register(myCounter); debugLevelRegistry.register(myCo

Re: [prometheus-users] Metrics vs log level

> Say, If we write a wrapper on top of prometheus java client API, its going to be messy You can make it relatively clean by creating (and incrementing) all the metrics, but only calling .register() on those that you want to expose in the given environment. Even more elaborately, you could have s

Re: [prometheus-users] Metrics vs log level

If you want to filter out some metrics, and you can't control this on the exporter, then you can use metric relabelling to drop any that you don't want to ingest into the database. On Friday, 7 O

Re: [prometheus-users] Metrics vs log level

On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote: we have a situation , where we have 8 to 15 million head series in each Prometheus and we have 7 instance of them (federated). Our prometheus are in a constant flooded situation handling the incoming metrics and back end recording rules.

[prometheus-users] Metrics vs log level

Hi Team, we have a situation , where we have 8 to 15 million head series in each Prometheus and we have 7 instance of them (federated). Our prometheus are in a constant flooded situation handling the incoming metrics and back end recording rules. One thought which came to was - do we have s