I'm currently on-boarding many more metrics and hosts to our Prometheus infrastructure and I wanted to know when is it advised to shard out metrics into a separate instance? I have multiple shards (or groups) of Prometheus instances although one of the groups of instances has over 3 million total series in it, even though its ingestion rate doesn't really go above a comfortable 100k series/second. I know that a single instance can typically handle up to 1 million series per second, although I can't seem to find clear guidelines with regards to the total number of series per Prometheus host. Some of the metrics have a relatively higher cardinality, thus causing the higher number of series. I'd appreciate any advice you can provide me with.
Thanks! -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/608224ac-f7fc-4dbb-9b18-0e7ed4f6e604%40googlegroups.com.
