Try increasing `capacity` to 3x max_samples_per_send, i.e. to 20000 for your case according to https://prometheus.io/docs/practices/remote_write/ .
Prometheus may require up to 30% more memory after enabling remote_write according to production measurements. Make sure that your Prometheus instance runs on a host with at least 30% of free memory before enabling remote_write on it. On Fri, Jan 15, 2021 at 11:18 AM Olga Chukanova <[email protected]> wrote: > Hello! > I have prometheus like monitoring system in kubernetes, and I trying to > set up remote_write to victoria metrics. But I have one tragic problem - my > prometheus dies by OOM. > I’ve tested two versions of Prometheus (v2.11.0 and v.2.23.0) and had > same problem on both. > My average value of rate(prometheus_remote_storage_samples_in_total [5m]) > is ~75k, prometheus pod limits is cpu ‘4’ and memory 6144M and average > metric prometheus_remote_storage_shards = 1. > Settings in remote_write are: > queue_config: > capacity: 100 > max_samples_per_send: 10000 > max_shards: 10 > min_shards: 1 > Global scrape setting: > global: > scrape_interval: 10s > scrape_timeout: 10s > evaluation_interval: 10s > In logs (with debug mode) I didn’t found anything, what can explayn the > problem. > I think, I’m doing something wrong in remote_write setting, but I don’t > understand what, and based on wich metrics I should configure that. > Thank you for any help! > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/830a40c0-dac9-43ca-880d-5898a2d70f50n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/830a40c0-dac9-43ca-880d-5898a2d70f50n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Best Regards, Aliaksandr Valialkin, CTO VictoriaMetrics -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPbKnmDJMGmBtbRZ%3DGXq3X7Du6MEtTvW_Ng07Gyfw8orU5GT6A%40mail.gmail.com.
