> why the memory keeps going up In your pictures from 24th November, I see mostly flat lines across the chart. "keeps going up" would be more like a slope.
> e.g. from 1.8G in avg to 2.3G I think you're talking about RSS here? It helps to be very specific - name the metric, give the time. RSS is Linux's guess of how much your process is using. It's not something you can make great deductions from, without other information. The Go heap target is much more concrete. Bryan On Monday, 4 December 2023 at 10:14:27 UTC win...@gmail.com wrote: > Hi Ben, Bryan > > > Go is a garbage collected language, memory use is going to fluctuate > over time as Prometheus operates and GC happens. It's not an exact value > and never will be > > We don't expect Prometheus memory is *exactly* the same with value before > the restart but are confusing why the memory keeps going up e.g. from 1.8G > in avg to 2.3G even having the same time series in HEAD block. > > With the graphs I shared above, the time series in the HEAD block has been > truncated to the original value which is 500K around 09:00; the memory did > came down from 3G to 2G at that time, but shortly it went up to ~2.8G even > no changes in the number of time series scraping. > > I did not setup dashboards with automatic refresh; and I think this issue > is always reproducible. > > Regards, Vu > On Thursday, November 30, 2023 at 8:50:04 PM UTC+7 Ben Kochie wrote: > >> It is. You can clearly see in your graphs that the Go memstats goes back >> down to the prior level. >> >> Go is a garbage collected language, memory use is going to fluctuate over >> time as Prometheus operates and GC happens. It's not an exact value and >> never will be. >> >> Memory use also depends on queries that are run, since each query >> requires some memory allocation. If your users have dashboards open with >> refresh, this will increase the amount of memory use. >> >> There are GC options like `GOGC` that you can tune, but of course this >> will impact how often GC is run, which could negatively impact performance. >> >> On Thu, Nov 30, 2023 at 2:30 PM Vu Nguyen <win...@gmail.com> wrote: >> >>> I could see the number of time series are truncated to a half, from 1M >>> to 500K but the memory does not go back to the point before target got >>> restarted (1) >>> >>> If I keep restarting the target the second time, the number of time >>> series will jump up to 1M and will eventually drop to 500M after few HEAD >>> truncation cycles but the memory now could be even higher than the case 1 >>> above. >>> >>> What confusing me here is that Prometheus memory does not return to the >>> point before restarting the target even the time series in HEAD goes back >>> to 500K. >>> >>> On Thu, Nov 30, 2023 at 6:16 PM Bryan Boreham <bjbo...@gmail.com> wrote: >>> >>>> >>>> >>>> > On 30 Nov 2023, at 07:20, Vu Nguyen <win...@gmail.com> wrote: >>>> > >>>> > >>>> > Thank you very much for your supports. >>>> > >>>> > > are you still deleting the WAL? >>>> > >>>> > No, I did not delete WAL at all. What I did was restarting a pod that >>>> has 500K time series exposed. >>>> > >>>> >>>> Ah, if there is any label different (eg the pod name) then that creates >>>> new series. >>>> So this would be consistent with the number of series bouncing up to >>>> 1,000,000. >>>> Prometheus only clears stale series out of the head after compaction, >>>> so you see series and memory go up for a couple of hours then come down >>>> again. >>>> >>>> In other words this is standard behaviour. >>>> >>>> Bryan >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-use...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/CAF7KRaxGCRsfoW7Z7hTKwwx_vZzx9UddS9cKm-OztZG%3Duz90vw%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/CAF7KRaxGCRsfoW7Z7hTKwwx_vZzx9UddS9cKm-OztZG%3Duz90vw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ba6c0b68-acfa-43c6-821d-340a3f1d8060n%40googlegroups.com.
