There have been some small improvements to the WAL, reducing the amount needed for startup. But you will have to upgrade to the latest release. This will not help the current situation you have, as the improvements come after upgrade.
Either way, there is almost never any choice about allocating more memory when you're hitting an OOM. Prometheus always uses the minimum amount of memory it can for operating. If it needs more memory, you need to decrease your metric cardinality or increase the allocated memory. There's no magic fix. On Tue, Feb 2, 2021 at 3:44 PM hamisa <[email protected]> wrote: > thanks for the quick reply. > > is there any way for me to disable it, or at least reduce the amount of > memory it consumes? > or for a change reduce the data capacity that being loaded from the wal? > > thanks in advance. > On Tuesday, 2 February 2021 at 16:25:06 UTC+2 [email protected] wrote: > >> You will need to give the instance more memory in order to complete the >> recovery. >> >> On Tue, Feb 2, 2021 at 2:32 PM hamisa <[email protected]> wrote: >> >>> Hi there, >>> I'm running prometheus 2.17 on my openshift 3.9.43 cluster. >>> when the prometheus pod collapse (deleted, evicted ...) the newly >>> created pod start reading the wal segments in addiotion to run prometheus >>> again. >>> in that process the prometheus' pod container get into OOM state, and >>> recreated just afterwards which causes CrashLoopBack to the pod. >>> >>> is there any solution to this problem? >>> >>> thanks in advance. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/b967175e-9658-4d6e-ac27-bb62a5a2f70bn%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/b967175e-9658-4d6e-ac27-bb62a5a2f70bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/95931435-aab7-4859-94b1-aee7743841bfn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/95931435-aab7-4859-94b1-aee7743841bfn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmomgrKS7-Gzkiy2hDkJDF2cu-9SW%2BEvY_50w_qa2LAQ9A%40mail.gmail.com.
