Thanks a lot, much appreciated. These are great insights and suggestions. I'll see how I can best use this to rework this monitoring strategy.
On Monday, January 25, 2021 at 1:47:42 PM UTC+1 [email protected] wrote: > Hi, > > Generally most metrics-based systems won't be able to deal with that level > of cardinality, thus typically we would recommend against putting data like > customer IDs into label values (unless there are only a handful of > customers). Prometheus is for monitoring overall system health, but would > have trouble tracking behavior down to this level of detail. For looking > into details of individual customers, you will likely need logging and/or > tracing systems, unless you really cut down on the number of series per > customer and either shard your Prometheus setup heavily or use > a horizontally scalable Prometheus implementation like Cortex. > > Regards, > Julius > > On Mon, Jan 25, 2021 at 10:53 AM [email protected] < > [email protected]> wrote: > >> Hello, >> I'm curious how to best use Prometheus for *large scale use cases*: >> >> Let's say you've got a successful e-commerce platform, using Prometheus >> to track some ~300 metrics related to requests, failures and interactions >> at various points throughout the system. >> Now, your platform is very generic, and you can easily support 10,000 >> different e-commerce sites & customers. >> At this scale, Kubernetes spreads your application across ~100 pods. >> (There's no "session aware" ingress router, so any pod can support any >> customer.) >> >> *How would you handle these metrics?* >> Adding the 'customer' label to every metric results in 300(metrics) * >> 10000(customers) * 100(pods) = 300M timeseries, scraped every 15 seconds by >> default. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/cb9974bd-96ba-4bdd-8cf5-e28cbf8f587cn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/cb9974bd-96ba-4bdd-8cf5-e28cbf8f587cn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a21dddf3-cb4a-44a7-85c3-71c932b7fbf4n%40googlegroups.com.
