I guess my suggestion is that crashing isn’t exactly a user friendly response to something that can happen days or weeks after initial setup (if only because of data growth) and I suggest prometheus would more actively try to prevent it by dynamically shrinking retention for instance
On Mon, Nov 9, 2020 at 10:06 Laurent Dumont <[email protected]> wrote: > Your retention is 16 days and you filled the data volume in 13? > > I don't Prometheus will alarm for it's data partition by default. > > On Mon, Nov 9, 2020, 12:27 PM Laurent Demailly <[email protected]> > wrote: > >> *What did you do?* >> Setup using community helm chart (8Gi data volume, 15 days retention) >> >> *What did you expect to see?* >> Would run indefinitely without crashing >> >> *What did you see instead? Under which circumstances?* >> Entered crash loop after 13 days because the volume got full. >> >> I expect instead some warning that older data is being deleted sooner >> than configured retention. or an alarm. Not a crash loop. >> >> *Environment* >> k8s >> >> >> (https://github.com/prometheus/prometheus/issues/8159 apparently must be >> an faq or "not a bug") >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. > > >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/f2a6a48a-7ab7-4df7-a8cc-3c2126e5c5d9n%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/f2a6a48a-7ab7-4df7-a8cc-3c2126e5c5d9n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BZk9WTzvhzEJ3zyX2O%3D16KfU6Km5EY6n%2ByfKrQh8YodLYP8uQ%40mail.gmail.com.
