On Mon, Jun 22, 2020 at 9:12 PM Paul Dubuc <[email protected]> wrote:
> I have a question about this for some clarification. How does the > '--metric-interval' affect the cardinality of the metric data? Does it > have the same multplier effect that '--series-interval' does in that every > 120 seconds (default) a whole new set of unique metrics are generated on > top of those generated at the '--series-interval'? > Yes. > > Also you you know if setting these intervals to 0 would keep the changes > from taking effect? > No. Just set `--metric-interval` to the value exceeding your test duration if you need suppressing churn rate. Something like the following should work: --metric-interval=100000 > > Thanks. > > On Thursday, December 5, 2019 at 2:09:55 PM UTC-5, Aliaksandr Valialkin > wrote: >> >> Note that `avalanche` introduces high churn rate for time series, i.e. >> old time series are constantly substituted by new time series every >> `--series-interval` seconds. Default value for `--series-interval` is 60 >> seconds, i.e. every 60 seconds new time series are created. So for >> `--metrics-count=1000 --series-count=1000` case `avalanche` introduces new >> 1M time series every minute. In 30 minutes Prometheus scrapes 30M time >> series from `avalanche`. See also `--metric-interval` command-line flag, >> which has almost the same meaning as `--series-interval`. >> >> BTW, how much RAM is available for your Prometheus setup? >> >> On Thu, Dec 5, 2019 at 9:34 AM Rupesh Tripathi <[email protected]> >> wrote: >> >>> Hello Folks, >>> >>> I performed some load/stress tests on Prometheus, Please find the >>> details and outcome below. I observed that prometheus docker container >>> abruptly disappeared/crashed in few instances. Can someone please help us >>> explaining what are the limitation of prometheus in terms of number of >>> unique metrics with high cardinality data? >>> >>> Steps: >>> >>> >>> >>> 1. Start Avalanche for producing unique metrics >>> 1. docker run -d --net=host quay.io/freshtracks.io/avalanche >>> --metric-count=1000 --series-count=1000 --port=9001 >>> 2. This will create 1000 unique metrics name each with 1000 >>> unique tag values, overall 1000*1000 unique metrics. >>> 3. Avalanche runs on 9001 port by default but we can change by >>> providing port value. >>> >>> >>> 1. *Start prometheus docker container pointing to 9001/metrics to >>> fetch metrics from Avalanche * >>> 1. docker run -d --net=host --rm -v >>> $(pwd)/prometheus0_eu2.yml:/etc/prometheus/prometheus.yml -v >>> $(pwd)/prometheus0_eu2_data:/prometheus -u root --name >>> prometheus-0-eu2 >>> prom/prometheus --config.file=/etc/prometheus/prometheus.yml >>> --storage.tsdb.path=/prometheus --web.listen-address=:9092 >>> --web.enable-lifecycle --web.enable-admin-api >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Number of unique metrics >>> >>> Duration >>> >>> CPU Usage (Average) >>> >>> Memory Usage >>> >>> Issues/outcome >>> >>> 1000 * 1000 = 1000,000 (1000 unique metric names each with 1000 unique >>> label/tag values) >>> >>> 10 - 15 minutes >>> >>> 15.0-20.0% >>> >>> 95-99% >>> >>> The prometheus container crashes and stops abruptly after 15-30 minutes, >>> most likely due to out of memory. >>> >>> 100 * 1000 = 100,000 (100 unique metric names each with 1000 unique >>> label/tag values) >>> >>> 1-1.5 hours >>> >>> 15.0-20.0% >>> >>> 90-99%(Starts increasing slowly and after an hour it grows to 90 %+ >>> >>> Prometheus container stops abruptly after running for 3-4 hours. >>> >>> 100 * 100 = 10,000 (100 unique metric names each with 100 unique >>> label/tag values) >>> >>> 2 days >>> >>> 5.0-7.0% >>> >>> 25-28% >>> >>> Prometheus service continues to run without any issues. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/2063df2a-cf7f-4e2c-a679-4fcef7cf5c9c%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/2063df2a-cf7f-4e2c-a679-4fcef7cf5c9c%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Best Regards, >> >> Aliaksandr >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/ce7d967c-7f56-4ec2-b549-770a51191e0do%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/ce7d967c-7f56-4ec2-b549-770a51191e0do%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Best Regards, Aliaksandr Valialkin, CTO VictoriaMetrics -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPbKnmDbLeocB7FKCNd-x2-xzqvZ7EaB38iayE8JazGMJ0NpvQ%40mail.gmail.com.
