On 2020-06-17 08:23, Dom Prittie wrote:
Hi,
I have a Prometheus deployment for which we have only used size based
retention, which we have set to 2.5T. We regularly see unexpectedly
large blocks getting created, so that when we hit
storage.tsdb.retention.size we see a huge drop in the metrics
available for querying.
For instance we currently have a block which is 1.1T and contains data
from Mon 1 Jun 01:00:00 BST 2020 to Sun 14 Jun 13:00:00 BST 2020,
which is ~45% of our retention size!
I can see from the docs that "Compaction will create larger blocks up
to 10% of the retention time, or 31 days, whichever is smaller". I
expected this to mean 10% of whatever retention I set, but from what I
have seen it looks like this means the max block size will always be
31 days if you have storage.tsdb.retention.time=0s and are just using
size based retention.
We have elected to use sized based retention instead of time because
we are in the process of onboarding application exporters so the
number of days we can retain is decreasing frequently, but the amount
of space we have available is not. In this situation what would be the
best way to configure storage.tsdb.max-block-duration?
What is the exact issue you are seeing, as it sounds like the storage
usage is still within the 2.5T limit you set?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/58883d30f3a95acf353ee940396b0bd4%40Jahingo.com.
