On Sat, 23 Jan 2021 at 10:11, Ben Kochie <[email protected]> wrote: > Many of the issues are related to query-time performance. It depends a lot > on the total cardinality at query time. > > For example, we have a couple of metrics with high total cardinality due > to the number of label combinations. One histogram metric has about 100k > total metrics after you multiple the number of instances times the number > of buckets times the number of labels on those buckets. > > In order to produce graphs and SLI/SLO reporting, we run a pretty standard > `rate(histogram_bucket[1m])` recording rule. This rule evaluation takes 30 > seconds to complete. Thankfully Prometheus TSDB has isolation for queries, > so even if we have overlapping rule evaluations, we don't break anything. > But it's still quite CPU and memory expensive to run these rules. > > Another thing to note about high cardinality is regexp matching `=~` is a > full table scan. So every query has to query all values in the label index. > This can start to take a long time when you have 10s of thousands of values > in a single label. Say the full scan takes 10ms. >
> That's fine for a single instant query, but Prometheus evaluation is a bit > different. Each point on the graph is a unique instant evaluation. So if > you have a 500 data-point graph, that's now 5 seconds of query time spent > parsing labels. > The index lookup is only done once for given selector for a range (i.e. graph) query. The principle applies though for the evaluation process after that, you'd have 500 evaluations of say 100k series each with a rate() on 5 samples, taking the rough rule of thumb of 10M samples processed per second that'd be around 25s for the query. Brian > > Ingestion is typically the easy part, querying is the hard part. > > On Sat, Jan 23, 2021 at 12:28 AM Girish Aher <[email protected]> wrote: > >> Hello Folks, >> A relatively new prometheus adopter here. Howdy! >> >> I have been reading through the terrific articles on Prometheus 2.x TSDB >> here: >> https://ganeshvernekar.com/blog/prometheus-tsdb-queries/ >> >> As I understand, the effect of high label cardinality (>1000 values for a >> labelname) is high memory usage. Assuming I can scale up the memory, are >> there any other effects of high label cardinality that one needs to be >> aware of? >> >> Could the ingestion/query performance be affected due to high label >> cardinality (despite having plenty of memory)? I gather from above articles >> that a map[labename][]postingOffset is maintained in memory per block and >> for a match query, it requires iterating through the []postingOffset >> sequentially comparing label value. Would this contribute to high query >> latencies? >> >> fyi - we are doing 4 million unique time series today with an ingestion >> rate of about 135K/sec. Pretty happy so far with the performance! :) >> >> >> >> --Girish >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/972e5772-0001-4fd1-8906-383884c70d63n%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/972e5772-0001-4fd1-8906-383884c70d63n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CABbyFmoE_di%3D6GeknvqSQcFPn1Soz%3DijvFqDN54dJii2dWjMPA%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CABbyFmoE_di%3D6GeknvqSQcFPn1Soz%3DijvFqDN54dJii2dWjMPA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLo_2sW-pjNJio_P8u6QgOgAABYAPbtFbnCAoFEKGZ8J8g%40mail.gmail.com.
