Re: [prometheus-developers] Preventing Prometheus from running out of memory

While I believe we should be opened to different solutions to limit the damages of Prometheus going OOM, I just want to point out that metrics use cases are wider than visibility and alerting. For example, missing metrics can get you to miss autoscaling .., and hence be dangerous. I’d be

[prometheus-developers] Re: Correlation between memory used and timeseries

You may be interested in this talk I gave at PromCon recently, showing some detail around where the memory is used: https://youtu.be/vc5LgoiP_CA Bryan On Friday, 25 November 2022 at 12:13:10 UTC l.mi...@gmail.com wrote: > This: > go_memstats_alloc_bytes / prometheus_tsdb_head_series > > gives

[prometheus-developers] Re: Should labels.Labels.Hash() be consistent?

I talked to people about this at PromCon recently; the consensus was that Prometheus does not expect that Labels.Hash() is constant across runs of Prometheus. Possible issues: * The 'hashmod' function in relabelling: this uses a MD5 hash, separate from Labels.Hash(). * Alert.String() calls

Re: [prometheus-developers] rbac support for exporter-toolkit

On 28 Nov 07:27, Bryan Boreham wrote: > I see that kube-rbac-proxy both authenticates the caller and performs an > authorization request to check whether that caller is allowed. > > Given divided opinions, how about a separate library which implements the > feature, and a hook in

Re: [prometheus-developers] rbac support for exporter-toolkit

I see that kube-rbac-proxy both authenticates the caller and performs an authorization request to check whether that caller is allowed. Given divided opinions, how about a separate library which implements the feature, and a hook in prometheus/exporter-toolkit so that any similar library can

Re: [prometheus-developers] Preventing Prometheus from running out of memory

To add to "Nothing will break if things are missing". Prometheus alerts depend on the existence of data for alerting to work. If you're "just missing some data", Prometheus alerts will fail to fire. This is unacceptable and we make specific design trade-offs in order to avoid this situation. On

Re: [prometheus-developers] Preventing Prometheus from running out of memory

On Mon, Nov 28, 2022 at 1:40 PM l.mi...@gmail.com wrote: > I agree that this is not side-effect free. With the worst possible outcome > of sending bogus alerts. > But. > We're talking about metrics, so from my PoV the consequences range not > from "annoying to dangerous" but "annoying to more

Re: [prometheus-developers] rbac support for exporter-toolkit

On 28 Nov 12:45, Ben Kochie wrote: > Yes, build it in. We don't want to require sidecars for every exporter. I disagree with this, as this sidecar is only required in kubernetes environments. Baking it into the exporter toolkit would be a huge maintenance challenge: - from users - which version

Re: [prometheus-developers] rbac support for exporter-toolkit

instead of adding kube-rbac-proxy as a sidecar, integrating with Kubernetes ServiceAccount + RBAC could be done directly using the Kube api. On Mon, Nov 28, 2022 at 12:45 PM Ben Kochie wrote: > Yes, build it in. We don't want to require sidecars for every exporter. > > On Mon, Nov 28, 2022 at

Re: [prometheus-developers] Preventing Prometheus from running out of memory

I agree that this is not side-effect free. With the worst possible outcome of sending bogus alerts. But. We're talking about metrics, so from my PoV the consequences range not from "annoying to dangerous" but "annoying to more annoying". There's no real danger in missing metrics. Nothing will

Re: [prometheus-developers] Preventing Prometheus from running out of memory

My reaction is similar to that of Ben and Julien: if metrics within a target go partially missing, all kinds of expectations around aggregations, histograms, and other metric correlations are off in ways that you wouldn't expect in a normal Prometheus setup. Everyone expects full scrape failures,

Re: [prometheus-developers] rbac support for exporter-toolkit

Yes, build it in. We don't want to require sidecars for every exporter. On Mon, Nov 28, 2022 at 12:43 PM Stuart Clark wrote: > On 2022-11-28 11:40, Ben Kochie wrote: > > It depends on if the sidecar is with Prometheus or with the target. > > > > If it's with Prometheus, that's probably just a

Re: [prometheus-developers] rbac support for exporter-toolkit

On 2022-11-28 11:40, Ben Kochie wrote: It depends on if the sidecar is with Prometheus or with the target. If it's with Prometheus, that's probably just a docs update. If it's with every exporter, that's probably something we would want in the exporter-toolkit. But, my understanding was that

Re: [prometheus-developers] rbac support for exporter-toolkit

It depends on if the sidecar is with Prometheus or with the target. If it's with Prometheus, that's probably just a docs update. If it's with every exporter, that's probably something we would want in the exporter-toolkit. But, my understanding was that the typical thing here was to use mTLS

Re: [prometheus-developers] rbac support for exporter-toolkit

On 2022-11-28 11:01, Jesús Samitier wrote: Yeah, maybe add some documentation with example configurations. If it just some docs I don't see any issue? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe

Re: [prometheus-developers] rbac support for exporter-toolkit

Yeah, maybe add some documentation with example configurations. On Mon, Nov 28, 2022 at 11:59 AM Stuart Clark wrote: > On 2022-11-28 10:56, Jesús Samitier wrote: > > Hi > > > > The idea is to integrate kube-rbac-proxy to add an extra (and > > optional) security feature in a new exporter, so the

Re: [prometheus-developers] rbac support for exporter-toolkit

On 2022-11-28 10:56, Jesús Samitier wrote: Hi The idea is to integrate kube-rbac-proxy to add an extra (and optional) security feature in a new exporter, so the final user can rely on RBAC to assure that only Prometheus can scrape its metrics. This is something you get when you install

Re: [prometheus-developers] rbac support for exporter-toolkit

Hi The idea is to integrate kube-rbac-proxy to add an extra (and optional) security feature in a new exporter, so the final user can rely on RBAC to assure that only Prometheus can scrape its metrics. This is something you get when you install Prometheus in K8s using the official helm chart -

Re: [prometheus-developers] rbac support for exporter-toolkit

What is meant by "add rbac support to the exporter-toolkit " ? I have been using Kubernetes and Prometheus for many years; I know what RBAC is but cannot immediately see how an exporter would want to write code that references it. Bryan On

[prometheus-developers] Changing consensus on HTTP headers

Hello, We once again got a request last week regarding overwriting HTTP header. This is a frequent request, even if it does not seem to affect a lot of users. However, I have crafted a pull request that changes that consensus and makes HTTP headers configurable in the common HTTP client, with

Re: [prometheus-developers] Preventing Prometheus from running out of memory

On Sunday, 27 November 2022 at 19:25:12 UTC sup...@gmail.com wrote: > Soft / partial failure modes can be very hard problems to deal with. You > have to be a lot more careful to not end up missing partial failures. > Sure, partial failures modes can be *in general* a hard problem, but with