I have a question about this for some clarification. How does the '--metric-interval' affect the cardinality of the metric data? Does it have the same multplier effect that '--series-interval' does in that every 120 seconds (default) a whole new set of unique metrics are generated on top of those generated at the '--series-interval'?
Also you you know if setting these intervals to 0 would keep the changes from taking effect? Thanks. On Thursday, December 5, 2019 at 2:09:55 PM UTC-5, Aliaksandr Valialkin wrote: > > Note that `avalanche` introduces high churn rate for time series, i.e. old > time series are constantly substituted by new time series every > `--series-interval` seconds. Default value for `--series-interval` is 60 > seconds, i.e. every 60 seconds new time series are created. So for > `--metrics-count=1000 --series-count=1000` case `avalanche` introduces new > 1M time series every minute. In 30 minutes Prometheus scrapes 30M time > series from `avalanche`. See also `--metric-interval` command-line flag, > which has almost the same meaning as `--series-interval`. > > BTW, how much RAM is available for your Prometheus setup? > > On Thu, Dec 5, 2019 at 9:34 AM Rupesh Tripathi <[email protected] > <javascript:>> wrote: > >> Hello Folks, >> >> I performed some load/stress tests on Prometheus, Please find the details >> and outcome below. I observed that prometheus docker container abruptly >> disappeared/crashed in few instances. Can someone please help us explaining >> what are the limitation of prometheus in terms of number of unique metrics >> with high cardinality data? >> >> Steps: >> >> >> >> 1. Start Avalanche for producing unique metrics >> 1. docker run -d --net=host quay.io/freshtracks.io/avalanche >> --metric-count=1000 --series-count=1000 --port=9001 >> 2. This will create 1000 unique metrics name each with 1000 unique >> tag values, overall 1000*1000 unique metrics. >> 3. Avalanche runs on 9001 port by default but we can change by >> providing port value. >> >> >> 1. *Start prometheus docker container pointing to 9001/metrics to >> fetch metrics from Avalanche * >> 1. docker run -d --net=host --rm -v >> $(pwd)/prometheus0_eu2.yml:/etc/prometheus/prometheus.yml -v >> $(pwd)/prometheus0_eu2_data:/prometheus -u root --name >> prometheus-0-eu2 >> prom/prometheus --config.file=/etc/prometheus/prometheus.yml >> --storage.tsdb.path=/prometheus --web.listen-address=:9092 >> --web.enable-lifecycle --web.enable-admin-api >> >> >> >> >> >> >> >> >> >> Number of unique metrics >> >> Duration >> >> CPU Usage (Average) >> >> Memory Usage >> >> Issues/outcome >> >> 1000 * 1000 = 1000,000 (1000 unique metric names each with 1000 unique >> label/tag values) >> >> 10 - 15 minutes >> >> 15.0-20.0% >> >> 95-99% >> >> The prometheus container crashes and stops abruptly after 15-30 minutes, >> most likely due to out of memory. >> >> 100 * 1000 = 100,000 (100 unique metric names each with 1000 unique >> label/tag values) >> >> 1-1.5 hours >> >> 15.0-20.0% >> >> 90-99%(Starts increasing slowly and after an hour it grows to 90 %+ >> >> Prometheus container stops abruptly after running for 3-4 hours. >> >> 100 * 100 = 10,000 (100 unique metric names each with 100 unique >> label/tag values) >> >> 2 days >> >> 5.0-7.0% >> >> 25-28% >> >> Prometheus service continues to run without any issues. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/2063df2a-cf7f-4e2c-a679-4fcef7cf5c9c%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/2063df2a-cf7f-4e2c-a679-4fcef7cf5c9c%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Best Regards, > > Aliaksandr > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ce7d967c-7f56-4ec2-b549-770a51191e0do%40googlegroups.com.
