My typical recommendation is that individual targets should try and keep their sum total metrics under 10k. So all metric/label combinations per target.
On Fri, Sep 4, 2020 at 8:31 AM rs vas <[email protected]> wrote: > Thank you /MR for your input, very helpful! > > Any recommendation/standards on a single Time Series - can't go beyond > some number? For example: > metric_name(name, label1, label2, label3) > label1 can have - 10 different values > label 2 can have - 50 different values > label 3 can have - 100 different values > Total number of unique series generated from a single node for this is: > 500,000 from single node. > if we have 1000 node - total of this unique series would be: 50 > million(50,000,000) > > This looks like not a great design to have too much cardinality on the > label values. Basically number values on each label will have a greater > impact on overall Prometheus performance and availability. > > Is there any recommendation on - one single metric - labels combination - > should not go beyond a number? for example: a metric on a single node > should not generate 50000 unique timeseries on single scrape or something > like that? > > rs vas > > On Wed, Sep 2, 2020 at 11:49 PM Matthias Rampke <[email protected]> > wrote: > >> For a (conservative) guide see this article >> <https://www.robustperception.io/cardinality-is-key>. You have the right >> intuition – ~10 buckets is a good number. >> >> You *can* go higher if you use high-powered machines (mostly: lots of >> RAM) for Prometheus, but you will run into increasing problems as either >> the cardinality of a single label, or the cardinality of a metric, gets >> large. Some of these you can work around by aggressively pre-aggregating in >> recording rules; otherwise it will be impossible to graph any reasonable >> timeframe. >> >> /MR >> >> On Wed, Sep 2, 2020 at 7:38 AM rs vas <[email protected]> wrote: >> >>> We are seeing an issue when we have used and generated a histogram with >>> 69le's it created a unique time series per le. this metric has few labels >>> for example: method, operation, identifier, ip >>> >>> This single metric itself is generating a 17k unique time series from a >>> single host and from all hosts its total time series is totalling up to >>> 300k for this xxxxx_seconds_bucket series. >>> >>> I was reading some best practices on defining buckets, we will have to >>> consider SLO and all, but my questions are: >>> >>> - any good number we can cross when defining buckets for example not >>> to define more than 10 le's. >>> - any good number of max labels you can have on a single metric >>> - any good number on the cardinality of a single time series at a >>> given point? If I query the count(xxxxx_seconds_bucket) what is that >>> number >>> we should not cross? >>> >>> I have a feeling like it is going to kill the Prometheus at some point, >>> if we can't fix the number of le's. Any input is appreciated. >>> >>> >>> *Example defined buckets: (looks to be a real concern here having 69 >>> le's).* >>> xxxxx_seconds_bucket{...="none",le="0.001",} 71849.0 >>> xxxxx_seconds_bucket{...="none",le="0.001048576",} 72078.0 >>> xxxxx_seconds_bucket{...="none",le="0.001398101",} 73083.0 >>> xxxxx_seconds_bucket{...="none",le="0.001747626",} 73600.0 >>> xxxxx_seconds_bucket{...="none",le="0.002097151",} 73943.0 >>> xxxxx_seconds_bucket{...="none",le="0.002446676",} 74160.0 >>> xxxxx_seconds_bucket{...="none",le="0.002796201",} 74399.0 >>> xxxxx_seconds_bucket{...="none",le="0.003145726",} 74936.0 >>> xxxxx_seconds_bucket{...="none",le="0.003495251",} 75109.0 >>> xxxxx_seconds_bucket{...="none",le="0.003844776",} 75227.0 >>> xxxxx_seconds_bucket{...="none",le="0.004194304",} 75336.0 >>> xxxxx_seconds_bucket{...="none",le="0.005592405",} 75674.0 >>> xxxxx_seconds_bucket{...="none",le="0.006990506",} 75885.0 >>> xxxxx_seconds_bucket{...="none",le="0.008388607",} 75958.0 >>> xxxxx_seconds_bucket{...="none",le="0.009786708",} 75981.0 >>> xxxxx_seconds_bucket{...="none",le="0.011184809",} 75995.0 >>> xxxxx_seconds_bucket{...="none",le="0.01258291",} 76004.0 >>> xxxxx_seconds_bucket{...="none",le="0.013981011",} 76005.0 >>> xxxxx_seconds_bucket{...="none",le="0.015379112",} 76008.0 >>> xxxxx_seconds_bucket{...="none",le="0.016777216",} 76013.0 >>> xxxxx_seconds_bucket{...="none",le="0.022369621",} 76033.0 >>> xxxxx_seconds_bucket{...="none",le="0.027962026",} 76039.0 >>> xxxxx_seconds_bucket{...="none",le="0.033554431",} 76039.0 >>> xxxxx_seconds_bucket{...="none",le="0.039146836",} 76039.0 >>> xxxxx_seconds_bucket{...="none",le="0.044739241",} 76039.0 >>> xxxxx_seconds_bucket{...="none",le="0.050331646",} 76039.0 >>> xxxxx_seconds_bucket{...="none",le="0.055924051",} 76039.0 >>> xxxxx_seconds_bucket{...="none",le="0.061516456",} 76042.0 >>> xxxxx_seconds_bucket{...="none",le="0.067108864",} 76043.0 >>> xxxxx_seconds_bucket{...="none",le="0.089478485",} 76044.0 >>> xxxxx_seconds_bucket{...="none",le="0.111848106",} 76044.0 >>> xxxxx_seconds_bucket{...="none",le="0.134217727",} 76044.0 >>> xxxxx_seconds_bucket{...="none",le="0.156587348",} 76044.0 >>> xxxxx_seconds_bucket{...="none",le="0.178956969",} 76044.0 >>> xxxxx_seconds_bucket{...="none",le="0.20132659",} 76045.0 >>> xxxxx_seconds_bucket{...="none",le="0.223696211",} 76045.0 >>> xxxxx_seconds_bucket{...="none",le="0.246065832",} 76046.0 >>> xxxxx_seconds_bucket{...="none",le="0.268435456",} 76046.0 >>> xxxxx_seconds_bucket{...="none",le="0.357913941",} 76046.0 >>> xxxxx_seconds_bucket{...,le="0.447392426",} 76057.0 >>> xxxxx_seconds_bucket{...,le="0.536870911",} 76061.0 >>> xxxxx_seconds_bucket{...,le="0.626349396",} 76061.0 >>> xxxxx_seconds_bucket{...,le="0.715827881",} 76064.0 >>> xxxxx_seconds_bucket{...,le="0.805306366",} 76085.0 >>> xxxxx_seconds_bucket{...,le="0.894784851",} 76085.0 >>> xxxxx_seconds_bucket{...,le="0.984263336",} 76086.0 >>> xxxxx_seconds_bucket{...,le="1.073741824",} 76086.0 >>> xxxxx_seconds_bucket{...,le="1.431655765",} 76086.0 >>> xxxxx_seconds_bucket{...,le="1.789569706",} 76088.0 >>> xxxxx_seconds_bucket{...,le="2.147483647",} 76116.0 >>> xxxxx_seconds_bucket{...,le="2.505397588",} 76116.0 >>> xxxxx_seconds_bucket{...,le="2.863311529",} 76116.0 >>> xxxxx_seconds_bucket{...,le="3.22122547",} 76116.0 >>> xxxxx_seconds_bucket{...,le="3.579139411",} 76116.0 >>> xxxxx_seconds_bucket{...,le="3.937053352",} 76116.0 >>> xxxxx_seconds_bucket{...,le="4.294967296",} 76116.0 >>> xxxxx_seconds_bucket{...,le="5.726623061",} 76116.0 >>> xxxxx_seconds_bucket{...,le="7.158278826",} 76116.0 >>> xxxxx_seconds_bucket{...,le="8.589934591",} 76116.0 >>> xxxxx_seconds_bucket{...,le="10.021590356",} 76116.0 >>> xxxxx_seconds_bucket{...,le="11.453246121",} 76116.0 >>> xxxxx_seconds_bucket{...,le="12.884901886",} 76116.0 >>> xxxxx_seconds_bucket{...,le="14.316557651",} 76116.0 >>> xxxxx_seconds_bucket{...,le="15.748213416",} 76116.0 >>> xxxxx_seconds_bucket{...,le="17.179869184",} 76116.0 >>> xxxxx_seconds_bucket{...,le="22.906492245",} 76116.0 >>> xxxxx_seconds_bucket{...,le="28.633115306",} 76116.0 >>> xxxxx_seconds_bucket{...,le="30.0",} 76116.0 >>> xxxxx_seconds_bucket{...,le="+Inf",} 76116.0 >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjxPf5mOLb1s%2BtgFXfjAqcZ7uOyMT1WrX3n3KymYNp_4Q%40mail.gmail.com >>> <https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjxPf5mOLb1s%2BtgFXfjAqcZ7uOyMT1WrX3n3KymYNp_4Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjdtTfYxUN8tCBpVhT-tzoj4H%2BOmegQFd6hfWKJWU0PPA%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjdtTfYxUN8tCBpVhT-tzoj4H%2BOmegQFd6hfWKJWU0PPA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqw3M%2BY0Jbe%3DNSbtv1R4haO37uHmXEgcYX1BniP_DUa1w%40mail.gmail.com.
