Issue : Prometheus Restarting with below error
``` level=error ts=2020-04-27T00:09:41.712Z caller=main.go:740
err="opening storage failed: block dir:
\"/prometheus/01DZCNZNP4SB2VVD8EY1V7HYSJ\": open
/prometheus/01DZCNZNP4SB2VVD8EY1V7HYSJ/meta.json: no such file or
directory" ```
Observation : 1. Checked the Datablocks & findout we lost around 300GB of
datablocks.
2. We have one disk space alert , Assuming that disk
got full before invoking this issue and something trigger the deleting on
old datablock the disk got free.
3. Moved the corrupted datablocks to /tmp directory
so promethus start working but old metrics are lost & After start its using
high resource 2X CPU load & high Memory utilisation.
Q1. Any Idea Why this error got triggered ?
Q2. What is the Normal behaviour of Promethus if no disk space is
available for storing the metrics ?
Q3 . If its deleted automatically then any parameters are available to
tweak this configurations ?
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/45855900-e87e-43a0-ae18-9ae50a14c88b%40googlegroups.com.
