Thank you /MR for your input, very helpful! Any recommendation/standards on a single Time Series - can't go beyond some number? For example: metric_name(name, label1, label2, label3) label1 can have - 10 different values label 2 can have - 50 different values label 3 can have - 100 different values Total number of unique series generated from a single node for this is: 500,000 from single node. if we have 1000 node - total of this unique series would be: 50 million(50,000,000)
This looks like not a great design to have too much cardinality on the label values. Basically number values on each label will have a greater impact on overall Prometheus performance and availability. Is there any recommendation on - one single metric - labels combination - should not go beyond a number? for example: a metric on a single node should not generate 50000 unique timeseries on single scrape or something like that? rs vas On Wed, Sep 2, 2020 at 11:49 PM Matthias Rampke <[email protected]> wrote: > For a (conservative) guide see this article > <https://www.robustperception.io/cardinality-is-key>. You have the right > intuition – ~10 buckets is a good number. > > You *can* go higher if you use high-powered machines (mostly: lots of > RAM) for Prometheus, but you will run into increasing problems as either > the cardinality of a single label, or the cardinality of a metric, gets > large. Some of these you can work around by aggressively pre-aggregating in > recording rules; otherwise it will be impossible to graph any reasonable > timeframe. > > /MR > > On Wed, Sep 2, 2020 at 7:38 AM rs vas <[email protected]> wrote: > >> We are seeing an issue when we have used and generated a histogram with >> 69le's it created a unique time series per le. this metric has few labels >> for example: method, operation, identifier, ip >> >> This single metric itself is generating a 17k unique time series from a >> single host and from all hosts its total time series is totalling up to >> 300k for this xxxxx_seconds_bucket series. >> >> I was reading some best practices on defining buckets, we will have to >> consider SLO and all, but my questions are: >> >> - any good number we can cross when defining buckets for example not >> to define more than 10 le's. >> - any good number of max labels you can have on a single metric >> - any good number on the cardinality of a single time series at a >> given point? If I query the count(xxxxx_seconds_bucket) what is that >> number >> we should not cross? >> >> I have a feeling like it is going to kill the Prometheus at some point, >> if we can't fix the number of le's. Any input is appreciated. >> >> >> *Example defined buckets: (looks to be a real concern here having 69 >> le's).* >> xxxxx_seconds_bucket{...="none",le="0.001",} 71849.0 >> xxxxx_seconds_bucket{...="none",le="0.001048576",} 72078.0 >> xxxxx_seconds_bucket{...="none",le="0.001398101",} 73083.0 >> xxxxx_seconds_bucket{...="none",le="0.001747626",} 73600.0 >> xxxxx_seconds_bucket{...="none",le="0.002097151",} 73943.0 >> xxxxx_seconds_bucket{...="none",le="0.002446676",} 74160.0 >> xxxxx_seconds_bucket{...="none",le="0.002796201",} 74399.0 >> xxxxx_seconds_bucket{...="none",le="0.003145726",} 74936.0 >> xxxxx_seconds_bucket{...="none",le="0.003495251",} 75109.0 >> xxxxx_seconds_bucket{...="none",le="0.003844776",} 75227.0 >> xxxxx_seconds_bucket{...="none",le="0.004194304",} 75336.0 >> xxxxx_seconds_bucket{...="none",le="0.005592405",} 75674.0 >> xxxxx_seconds_bucket{...="none",le="0.006990506",} 75885.0 >> xxxxx_seconds_bucket{...="none",le="0.008388607",} 75958.0 >> xxxxx_seconds_bucket{...="none",le="0.009786708",} 75981.0 >> xxxxx_seconds_bucket{...="none",le="0.011184809",} 75995.0 >> xxxxx_seconds_bucket{...="none",le="0.01258291",} 76004.0 >> xxxxx_seconds_bucket{...="none",le="0.013981011",} 76005.0 >> xxxxx_seconds_bucket{...="none",le="0.015379112",} 76008.0 >> xxxxx_seconds_bucket{...="none",le="0.016777216",} 76013.0 >> xxxxx_seconds_bucket{...="none",le="0.022369621",} 76033.0 >> xxxxx_seconds_bucket{...="none",le="0.027962026",} 76039.0 >> xxxxx_seconds_bucket{...="none",le="0.033554431",} 76039.0 >> xxxxx_seconds_bucket{...="none",le="0.039146836",} 76039.0 >> xxxxx_seconds_bucket{...="none",le="0.044739241",} 76039.0 >> xxxxx_seconds_bucket{...="none",le="0.050331646",} 76039.0 >> xxxxx_seconds_bucket{...="none",le="0.055924051",} 76039.0 >> xxxxx_seconds_bucket{...="none",le="0.061516456",} 76042.0 >> xxxxx_seconds_bucket{...="none",le="0.067108864",} 76043.0 >> xxxxx_seconds_bucket{...="none",le="0.089478485",} 76044.0 >> xxxxx_seconds_bucket{...="none",le="0.111848106",} 76044.0 >> xxxxx_seconds_bucket{...="none",le="0.134217727",} 76044.0 >> xxxxx_seconds_bucket{...="none",le="0.156587348",} 76044.0 >> xxxxx_seconds_bucket{...="none",le="0.178956969",} 76044.0 >> xxxxx_seconds_bucket{...="none",le="0.20132659",} 76045.0 >> xxxxx_seconds_bucket{...="none",le="0.223696211",} 76045.0 >> xxxxx_seconds_bucket{...="none",le="0.246065832",} 76046.0 >> xxxxx_seconds_bucket{...="none",le="0.268435456",} 76046.0 >> xxxxx_seconds_bucket{...="none",le="0.357913941",} 76046.0 >> xxxxx_seconds_bucket{...,le="0.447392426",} 76057.0 >> xxxxx_seconds_bucket{...,le="0.536870911",} 76061.0 >> xxxxx_seconds_bucket{...,le="0.626349396",} 76061.0 >> xxxxx_seconds_bucket{...,le="0.715827881",} 76064.0 >> xxxxx_seconds_bucket{...,le="0.805306366",} 76085.0 >> xxxxx_seconds_bucket{...,le="0.894784851",} 76085.0 >> xxxxx_seconds_bucket{...,le="0.984263336",} 76086.0 >> xxxxx_seconds_bucket{...,le="1.073741824",} 76086.0 >> xxxxx_seconds_bucket{...,le="1.431655765",} 76086.0 >> xxxxx_seconds_bucket{...,le="1.789569706",} 76088.0 >> xxxxx_seconds_bucket{...,le="2.147483647",} 76116.0 >> xxxxx_seconds_bucket{...,le="2.505397588",} 76116.0 >> xxxxx_seconds_bucket{...,le="2.863311529",} 76116.0 >> xxxxx_seconds_bucket{...,le="3.22122547",} 76116.0 >> xxxxx_seconds_bucket{...,le="3.579139411",} 76116.0 >> xxxxx_seconds_bucket{...,le="3.937053352",} 76116.0 >> xxxxx_seconds_bucket{...,le="4.294967296",} 76116.0 >> xxxxx_seconds_bucket{...,le="5.726623061",} 76116.0 >> xxxxx_seconds_bucket{...,le="7.158278826",} 76116.0 >> xxxxx_seconds_bucket{...,le="8.589934591",} 76116.0 >> xxxxx_seconds_bucket{...,le="10.021590356",} 76116.0 >> xxxxx_seconds_bucket{...,le="11.453246121",} 76116.0 >> xxxxx_seconds_bucket{...,le="12.884901886",} 76116.0 >> xxxxx_seconds_bucket{...,le="14.316557651",} 76116.0 >> xxxxx_seconds_bucket{...,le="15.748213416",} 76116.0 >> xxxxx_seconds_bucket{...,le="17.179869184",} 76116.0 >> xxxxx_seconds_bucket{...,le="22.906492245",} 76116.0 >> xxxxx_seconds_bucket{...,le="28.633115306",} 76116.0 >> xxxxx_seconds_bucket{...,le="30.0",} 76116.0 >> xxxxx_seconds_bucket{...,le="+Inf",} 76116.0 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjxPf5mOLb1s%2BtgFXfjAqcZ7uOyMT1WrX3n3KymYNp_4Q%40mail.gmail.com >> <https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjxPf5mOLb1s%2BtgFXfjAqcZ7uOyMT1WrX3n3KymYNp_4Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPs_AfjdtTfYxUN8tCBpVhT-tzoj4H%2BOmegQFd6hfWKJWU0PPA%40mail.gmail.com.
