Such a short retention is unlikely to help at all; WAL blocks have a 2 hour
duration I think.
Across some systems I have here, the average number of metrics per node is
2366: this is the (expensive) query which gives it:
avg(count by (instance) ({job="node"}))
So with 1300 nodes that would be about 3 million metrics. Quite a lot, but
not extraordinarily so. I've seen recommendations to start splitting
Prometheus servers when you reach 2m. There is a RAM calculation tool here:
https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion
With 3m series and 1m unique label pairs, it still only comes out to 8GB.
If you're needing much more than that, then you need to read and understand
the stats from the TSDB status page. You can post them here if you want
help interpreting them. And you need to understand what queries (if any)
are taking place against your database, since those use RAM too.
Looking at "Top 10 series count by metric names" in the Prometheus Status
page, in my case it's node_cpu_seconds_total{}. For me it's
node_cpu_seconds_total{}. If you don't require the usage of each core
individually, then you might be inclined to drop it.
You could also see if victoriametrics + vmagent works better for your use
case.
On Tuesday, 7 September 2021 at 13:57:48 UTC+1 [email protected] wrote:
> Thank you Brian for the reply. Yes I mean host (nodes).
> What we have done for the mean time is we have set the retentionTime of
> prometheus to 5minutes (which I am not comfortable) but was advised by
> seniors just for us to continue.
> Thanks for the information above, i'll check it out and try on our cluster
> environment.
>
>
>
>
>
>
> On Tue, Sep 7, 2021 at 4:50 PM Brian Candler <[email protected]> wrote:
>
>> It's not clear what you mean by "No. of Nodes" - whether you mean hosts
>> (e.g. which you're scraping using node_exporter), or pods, or something
>> else. But what matters is the total number of metrics, and the amount of
>> metric churn, i.e. the rate at which new timeseries are being created
>> dynamically; and also how much querying is going on.
>>
>> If you go to Prometheus web interface, Status > TSDB Status, you'll get
>> some statistics which may help you. Consider:
>>
>> - collecting fewer metrics (by changing what you scrape, and/or using
>> metric_relabel_configs to drop some timeseries which are not of interest)
>>
>> - see if it's possible to reduce timeseries churn. For example, if you
>> have one application which is generating large numbers of short-lived pods
>> then you may wish to reduce or suppress the metrics collected for those
>> pods.
>>
>> - have a look at the PromQL queries being executed, and whether any of
>> these are using excessing amounts of RAM. The query log
>> <https://prometheus.io/docs/guides/query-log/> may help. You can also
>> apply limits to how much memory is used by individual queries using
>> --query.max-concurrency=20 # default
>> --query.max-samples=50000000 # default
>> (although that may cause the offending queries to fail)
>>
>> There are also blog posts out there which you can turn up with a search,
>> e.g.
>> https://source.coveo.com/2021/03/03/prometheus-memory/
>>
>> On Tuesday, 7 September 2021 at 07:34:51 UTC+1 [email protected] wrote:
>>
>>> Hi everyone, I am new here.
>>>
>>> I would like to seek some advice on the design approach we should take.
>>> With the given problem below, in terms of cost, how can we set up
>>> Prometheus with a large cluster.
>>>
>>> *Variables:*
>>> *Installation: *Kube-stack-prometheus helm chart.
>>> *Autoscale*: yes
>>> *No. of Nodes*: 1000 up to 1300
>>> *Mesh*: Istio
>>> *Memory Usage:* 50GB (Still gets OOM)
>>> *Installed: *1 Prometheus, 1 Kiali, 1 Grafana and 1 Jaeger
>>>
>>> *Issue:*
>>> 1. We cannot expand a larger node for Prometheus as 60GB memory is
>>> already expensive. (cost not approved by management)
>>> 2. Removing unnecessary metrics is not yet advised because we do not
>>> know which metrics of istio, jaeger and kiali are needed.
>>>
>>> *Tried solution:*
>>> We have federated the single instance of prometheus with Thanos
>>> Receivers, however, the issue is still there because kiali queries its data
>>> directly from prometheus which eventually gets OOM.
>>>
>>> *Question:*
>>> We are thinking of firing up multiple prometheus for each namespace and
>>> adding thanos-sidecar with the same scrape config since thanos will
>>> deduplicate all duplicated metrics. This approach would solve the issue in
>>> Grafana queries but not in Kiali.
>>>
>>> How can we set up a multiple prometheus (low cost) but single instance
>>> prometheus for kiali (whole cluster)?
>>>
>>> Appreciate any help. Thank you.
>>>
>>>
>>>
>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/24a15533-094e-4a4c-9644-5d4375b6aaa2n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/24a15533-094e-4a4c-9644-5d4375b6aaa2n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bde269c1-119e-4d1e-a899-9f27332b0ff6n%40googlegroups.com.
