I do think we need a better way to detect when we are overloaded,
especially with respect to memory usage, and have defined behavior for how
we handle backpressure in those cases. The current experience of entering
OOM loops is frustrating, and makes it hard to debug as you can't query
anything to
The problem is not that much priorities etc, it is all the questions and
confusions around this:
- When do we decide we are overloaded?
- What do we do for the low priority targets?
and more importantly:
- When do we decide that we can scrape the low targets again?
How to avoid:
High load ->
Yes, looks like having many scrapers would solve this, and having Thanos on
top for query aggregation can do. However, given the overhead of even
operating the TSDB instances like Prometheus (e.g maintaining persistence
volumes), I would still see some longer-term solution of better multitenant
Thanks, everyone for the replies! The official msg seems to be to use a
Prometheus instance per tenant/priority if you want to have multiple
tenants in your environment.
Kind regards,
Lili
On Thursday, 30 July 2020 10:44:59 UTC+2, Ben Kochie wrote:
>
> I'm with Brian and Julian on this.
>
>
I'm with Brian and Julian on this.
Multi-tenancy is not really something we want to solve in Prometheus. This
is a concern for higher level systems like Kubernetes. Prometheus is
designed to be distributed. If you have targets with different needs, they
need to have separate Prometheus instances.
That's only effective in limiting the number of targets, the point here is
that selectively scraping those with a higher priority based on
backpressure of the system as a whole.
On Wed, 22 Jul 2020 at 17:00, Julien Pivotto
wrote:
> On 22 Jul 16:47, Frederic Branczyk wrote:
> > In practice even
On 22 Jul 16:47, Frederic Branczyk wrote:
> In practice even that can still be problematic. You only know that
> Prometheus has a problem when everything fails, the point is to keep things
> alive well enough for more critical components.
>
> On Wed, 22 Jul 2020 at 16:38, Julien Pivotto
> wrote:
On 22 Jul 16:36, Frederic Branczyk wrote:
> It's unclear how that helps, can you help me understand?
- job: highprio
relabel_configs:
- target_label: job
replacement: pods
- source_labels: [__meta_pod_priority]
regex: high
action: keep
- job: lowprio
relabel_configs:
-
It's unclear how that helps, can you help me understand?
On Wed, 22 Jul 2020 at 16:34, Julien Pivotto
wrote:
> On 22 Jul 16:32, Frederic Branczyk wrote:
> > Can you explain what you mean by two jobs? Do you mean two scrape
> configs?
>
> Yes.
>
> >
> > On Wed, 22 Jul 2020 at 11:40, Julien
On 22 Jul 16:32, Frederic Branczyk wrote:
> Can you explain what you mean by two jobs? Do you mean two scrape configs?
Yes.
>
> On Wed, 22 Jul 2020 at 11:40, Julien Pivotto
> wrote:
>
> > On 22 Jul 02:35, Lili Cosic wrote:
> > >
> > >
> > > On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian
Can you explain what you mean by two jobs? Do you mean two scrape configs?
On Wed, 22 Jul 2020 at 11:40, Julien Pivotto
wrote:
> On 22 Jul 02:35, Lili Cosic wrote:
> >
> >
> > On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil wrote:
> > >
> > > On Wed, 22 Jul 2020 at 10:18, Julien Pivotto
On 22 Jul 02:35, Lili Cosic wrote:
>
>
> On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil wrote:
> >
> > On Wed, 22 Jul 2020 at 10:18, Julien Pivotto > > wrote:
> >
> >> On 22 Jul 02:14, Lili Cosic wrote:
> >> > Only now seen in the docs that I am supposed to start any discussions
> >>
On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil wrote:
>
> On Wed, 22 Jul 2020 at 10:18, Julien Pivotto > wrote:
>
>> On 22 Jul 02:14, Lili Cosic wrote:
>> > Only now seen in the docs that I am supposed to start any discussions
>> here
>> > first before opening an issue, sorry about
Only now seen in the docs that I am supposed to start any discussions here
first before opening an issue, sorry about that! :)
Currently there is no way of a target to have higher scrape priority over
another, but if you have a setup and even if you set target limits and
sample limits you can
14 matches
Mail list logo
