Firstly - thank you for your answer and advice! I've increased capacity to 20000, but it didn't help. Anyway, I had to do it, I misread the documentation, thank you. To calculate required memory, I used this calc: https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion With my max_over_time(prometheus_tsdb_head_series[1d]) = 769255 and unique label pairs (count(kube_pod_labels{app=~".*"})) 1154, Combined Memory = 3,868MiB Plus 30% for remote_write and we have ~5.1MiB My pod has 6Gb memory in limits. May it be not enough 1Gb in reserve? I know if I increase the memory limit to 8Gb it can solve my problem, I just want to be sure this is only one way.
пт, 15 янв. 2021 г. в 13:52, Aliaksandr Valialkin <[email protected]>: > Try increasing `capacity` to 3x max_samples_per_send, i.e. to 20000 for > your case according to https://prometheus.io/docs/practices/remote_write/ > . > > Prometheus may require up to 30% more memory after enabling remote_write > according to production measurements. Make sure that your Prometheus > instance runs on a host with at least 30% of free memory before enabling > remote_write on it. > > On Fri, Jan 15, 2021 at 11:18 AM Olga Chukanova <[email protected]> > wrote: > >> Hello! >> I have prometheus like monitoring system in kubernetes, and I trying to >> set up remote_write to victoria metrics. But I have one tragic problem - my >> prometheus dies by OOM. >> I’ve tested two versions of Prometheus (v2.11.0 and v.2.23.0) and had >> same problem on both. >> My average value of rate(prometheus_remote_storage_samples_in_total [5m]) >> is ~75k, prometheus pod limits is cpu ‘4’ and memory 6144M and average >> metric prometheus_remote_storage_shards = 1. >> Settings in remote_write are: >> queue_config: >> capacity: 100 >> max_samples_per_send: 10000 >> max_shards: 10 >> min_shards: 1 >> Global scrape setting: >> global: >> scrape_interval: 10s >> scrape_timeout: 10s >> evaluation_interval: 10s >> In logs (with debug mode) I didn’t found anything, what can explayn the >> problem. >> I think, I’m doing something wrong in remote_write setting, but I don’t >> understand what, and based on wich metrics I should configure that. >> Thank you for any help! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/830a40c0-dac9-43ca-880d-5898a2d70f50n%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/830a40c0-dac9-43ca-880d-5898a2d70f50n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Best Regards, > > Aliaksandr Valialkin, CTO VictoriaMetrics > -- С уважением, Ольга. +7 (968) 760-89-95 -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAH81ULF1XnsojRaABO8L9L8tv9hsC4_nO%2BSdeCh9VBS3-3QPrA%40mail.gmail.com.
