I don't think Prometheus will handle time. But you can probably use --storage.tsdb.retention.size and have it slightly lower than the size of the actual data volume. That way, you won't fill up the volume and crash the server.
On Mon, Nov 9, 2020 at 8:15 PM Laurent Demailly <[email protected]> wrote: > I guess my suggestion is that crashing isn’t exactly a user friendly > response to something that can happen days or weeks after initial setup (if > only because of data growth) and I suggest prometheus would more actively > try to prevent it by dynamically shrinking retention for instance > > On Mon, Nov 9, 2020 at 10:06 Laurent Dumont <[email protected]> > wrote: > >> Your retention is 16 days and you filled the data volume in 13? >> >> I don't Prometheus will alarm for it's data partition by default. >> >> On Mon, Nov 9, 2020, 12:27 PM Laurent Demailly <[email protected]> >> wrote: >> >>> *What did you do?* >>> Setup using community helm chart (8Gi data volume, 15 days retention) >>> >>> *What did you expect to see?* >>> Would run indefinitely without crashing >>> >>> *What did you see instead? Under which circumstances?* >>> Entered crash loop after 13 days because the volume got full. >>> >>> I expect instead some warning that older data is being deleted sooner >>> than configured retention. or an alarm. Not a crash loop. >>> >>> *Environment* >>> k8s >>> >>> >>> (https://github.com/prometheus/prometheus/issues/8159 apparently must >>> be an faq or "not a bug") >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >> >> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/f2a6a48a-7ab7-4df7-a8cc-3c2126e5c5d9n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/f2a6a48a-7ab7-4df7-a8cc-3c2126e5c5d9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOAKi8xioW0KqFxq0H38c5ptaGsoDZxghy-VVuFzLo13BM_eeg%40mail.gmail.com.
