> I believe the inactive time series should be regularly flushed out of active memory. High cardinality caused by the high churn rate should not cause high RAM usage.
As I understand it, the full set of timeseries seen will remain in the "head chunk" for 2 hours. So high churn rate *does* cause high RAM usage. If you create and destroy 100 pods per minute, and each pod generates 1000 metrics, then in 2 hours that's 12 million timeseries. In several ways the stress on the TSDB is similar to scraping 12 million timeseries, even though in a particular scrape you'll only be ingesting data for a small subset of those. You may find these useful: https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality https://prometheus.io/docs/prometheus/latest/storage/ Also, this is an old document (and sadly no longer published at a stable URL), but it gives the design of the TSDB from when Prometheus 2.0 was created: https://web.archive.org/web/20180125094157/https://fabxc.org/tsdb/ -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ee68f123-1504-4d69-ab36-829f24fe3e85n%40googlegroups.com.
