[image: Capture-wal-corruption.PNG] /usr/local/bin/prometheus --version prometheus, version 2.20.0 (branch: HEAD, revision: e5a06b483527d4fe0704b8fa3a2b475b661c526f) build user: r build date: 20200722-18:51:45 go version: go1.14.6
sudo du -Pshx /var/lib/prometheus/* 73G /var/lib/prometheus/chunks_head 0 /var/lib/prometheus/lock 0 /var/lib/prometheus/queries.active 120G /var/lib/prometheus/wal Data partition was filled a some point, we increased it but since than noticing that wal partition is growing all the time. In the logs seeing *prometheus[2172]: level=info ts=2020-08-17T19:14:25.361Z caller=head.go:760 component=tsdb msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"*Also - seeing these errors: *level=error ts=2020-08-17T19:26:19.628Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: truncate chunks.HeadReadWriter: maxt of the files are not set"* -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ad16a232-be8a-4787-88bd-4580ef667db0n%40googlegroups.com.
