For a (conservative) guide see this article <https://www.robustperception.io/cardinality-is-key>. You have the right intuition – ~10 buckets is a good number.
You *can* go higher if you use high-powered machines (mostly: lots of RAM) for Prometheus, but you will run into increasing problems as either the cardinality of a single label, or the cardinality of a metric, gets large. Some of these you can work around by aggressively pre-aggregating in recording rules; otherwise it will be impossible to graph any reasonable timeframe. /MR On Wed, Sep 2, 2020 at 7:38 AM rs vas <[email protected]> wrote: > We are seeing an issue when we have used and generated a histogram with > 69le's it created a unique time series per le. this metric has few labels > for example: method, operation, identifier, ip > > This single metric itself is generating a 17k unique time series from a > single host and from all hosts its total time series is totalling up to > 300k for this xxxxx_seconds_bucket series. > > I was reading some best practices on defining buckets, we will have to > consider SLO and all, but my questions are: > > - any good number we can cross when defining buckets for example not > to define more than 10 le's. > - any good number of max labels you can have on a single metric > - any good number on the cardinality of a single time series at a > given point? If I query the count(xxxxx_seconds_bucket) what is that number > we should not cross? > > I have a feeling like it is going to kill the Prometheus at some point, if > we can't fix the number of le's. Any input is appreciated. > > > *Example defined buckets: (looks to be a real concern here having 69 > le's).* > xxxxx_seconds_bucket{...="none",le="0.001",} 71849.0 > xxxxx_seconds_bucket{...="none",le="0.001048576",} 72078.0 > xxxxx_seconds_bucket{...="none",le="0.001398101",} 73083.0 > xxxxx_seconds_bucket{...="none",le="0.001747626",} 73600.0 > xxxxx_seconds_bucket{...="none",le="0.002097151",} 73943.0 > xxxxx_seconds_bucket{...="none",le="0.002446676",} 74160.0 > xxxxx_seconds_bucket{...="none",le="0.002796201",} 74399.0 > xxxxx_seconds_bucket{...="none",le="0.003145726",} 74936.0 > xxxxx_seconds_bucket{...="none",le="0.003495251",} 75109.0 > xxxxx_seconds_bucket{...="none",le="0.003844776",} 75227.0 > xxxxx_seconds_bucket{...="none",le="0.004194304",} 75336.0 > xxxxx_seconds_bucket{...="none",le="0.005592405",} 75674.0 > xxxxx_seconds_bucket{...="none",le="0.006990506",} 75885.0 > xxxxx_seconds_bucket{...="none",le="0.008388607",} 75958.0 > xxxxx_seconds_bucket{...="none",le="0.009786708",} 75981.0 > xxxxx_seconds_bucket{...="none",le="0.011184809",} 75995.0 > xxxxx_seconds_bucket{...="none",le="0.01258291",} 76004.0 > xxxxx_seconds_bucket{...="none",le="0.013981011",} 76005.0 > xxxxx_seconds_bucket{...="none",le="0.015379112",} 76008.0 > xxxxx_seconds_bucket{...="none",le="0.016777216",} 76013.0 > xxxxx_seconds_bucket{...="none",le="0.022369621",} 76033.0 > xxxxx_seconds_bucket{...="none",le="0.027962026",} 76039.0 > xxxxx_seconds_bucket{...="none",le="0.033554431",} 76039.0 > xxxxx_seconds_bucket{...="none",le="0.039146836",} 76039.0 > xxxxx_seconds_bucket{...="none",le="0.044739241",} 76039.0 > xxxxx_seconds_bucket{...="none",le="0.050331646",} 76039.0 > xxxxx_seconds_bucket{...="none",le="0.055924051",} 76039.0 > xxxxx_seconds_bucket{...="none",le="0.061516456",} 76042.0 > xxxxx_seconds_bucket{...="none",le="0.067108864",} 76043.0 > xxxxx_seconds_bucket{...="none",le="0.089478485",} 76044.0 > xxxxx_seconds_bucket{...="none",le="0.111848106",} 76044.0 > xxxxx_seconds_bucket{...="none",le="0.134217727",} 76044.0 > xxxxx_seconds_bucket{...="none",le="0.156587348",} 76044.0 > xxxxx_seconds_bucket{...="none",le="0.178956969",} 76044.0 > xxxxx_seconds_bucket{...="none",le="0.20132659",} 76045.0 > xxxxx_seconds_bucket{...="none",le="0.223696211",} 76045.0 > xxxxx_seconds_bucket{...="none",le="0.246065832",} 76046.0 > xxxxx_seconds_bucket{...="none",le="0.268435456",} 76046.0 > xxxxx_seconds_bucket{...="none",le="0.357913941",} 76046.0 > xxxxx_seconds_bucket{...,le="0.447392426",} 76057.0 > xxxxx_seconds_bucket{...,le="0.536870911",} 76061.0 > xxxxx_seconds_bucket{...,le="0.626349396",} 76061.0 > xxxxx_seconds_bucket{...,le="0.715827881",} 76064.0 > xxxxx_seconds_bucket{...,le="0.805306366",} 76085.0 > xxxxx_seconds_bucket{...,le="0.894784851",} 76085.0 > xxxxx_seconds_bucket{...,le="0.984263336",} 76086.0 > xxxxx_seconds_bucket{...,le="1.073741824",} 76086.0 > xxxxx_seconds_bucket{...,le="1.431655765",} 76086.0 > xxxxx_seconds_bucket{...,le="1.789569706",} 76088.0 > xxxxx_seconds_bucket{...,le="2.147483647",} 76116.0 > xxxxx_seconds_bucket{...,le="2.505397588",} 76116.0 > xxxxx_seconds_bucket{...,le="2.863311529",} 76116.0 > xxxxx_seconds_bucket{...,le="3.22122547",} 76116.0 > xxxxx_seconds_bucket{...,le="3.579139411",} 76116.0 > xxxxx_seconds_bucket{...,le="3.937053352",} 76116.0 > xxxxx_seconds_bucket{...,le="4.294967296",} 76116.0 > xxxxx_seconds_bucket{...,le="5.726623061",} 76116.0 > xxxxx_seconds_bucket{...,le="7.158278826",} 76116.0 > xxxxx_seconds_bucket{...,le="8.589934591",} 76116.0 > xxxxx_seconds_bucket{...,le="10.021590356",} 76116.0 > xxxxx_seconds_bucket{...,le="11.453246121",} 76116.0 > xxxxx_seconds_bucket{...,le="12.884901886",} 76116.0 > xxxxx_seconds_bucket{...,le="14.316557651",} 76116.0 > xxxxx_seconds_bucket{...,le="15.748213416",} 76116.0 > xxxxx_seconds_bucket{...,le="17.179869184",} 76116.0 > xxxxx_seconds_bucket{...,le="22.906492245",} 76116.0 > xxxxx_seconds_bucket{...,le="28.633115306",} 76116.0 > xxxxx_seconds_bucket{...,le="30.0",} 76116.0 > xxxxx_seconds_bucket{...,le="+Inf",} 76116.0 > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjxPf5mOLb1s%2BtgFXfjAqcZ7uOyMT1WrX3n3KymYNp_4Q%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjxPf5mOLb1s%2BtgFXfjAqcZ7uOyMT1WrX3n3KymYNp_4Q%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbiZ%3DruT0e%3D%3DaHyJ5dLEYpLQd4XraubS7DWEFBu%3Drj%3DoQ%40mail.gmail.com.
