Background on why I wanted to play around this parameter: Using LTS version for testing i.e. 2.45.2 During compaction i.e. every 3days, the resident memory of prometheus spikes to a very high value. Example if average of process_resident_memory_bytes is around 50 GB and at the time of compaction it spikes to 120 to 160 GB. Considering the usage of 50 GB want memory allocated to the host to be around 128GB. But looking at memory usage spike during compaction, this doesn't seem to be a workable option and keeping a low value may lead to OOM during compaction. It also adds to cost for cloud based VMs. On Wednesday, January 24, 2024 at 1:35:16 PM UTC-8 Sukhada Sankpal wrote:
> storage.tsdb.max-block-duration default value is set to be 10% of > retention time. I am currently using a setup with 30 days of retention and > thereby this flags default value is set to be 3 days. > Based on suggestions posted here: > https://github.com/prometheus/prometheus/issues/6934#issuecomment-1610921555 > I changed storage.tsdb.min-block-duration to 30m and > storage.tsdb.max-block-duration to 1h. This resulted in no-compaction state > and local storage increased quickly. > > In order to enable the compaction and have a safe test, I changed > storage.tsdb.max-block-duration to 1day > > I want some guideline on what is a safe lower value of this parameter and > keeping it low impact in increased memory usage? > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3de4f70c-7344-4d19-a117-0951ca7733f8n%40googlegroups.com.
