I have a federated setup of Prometheus 2.15.2 which recently started to show gaps and throwing a lot of errors like:
caller=db.go:609 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: create checkpoint: read segments: corruption in segment /prometheus/wal/checkpoint.049194/00000000 at 96731136: unexpected non-zero byte in padded page" So I removed /prometheus/wal/checkpoint.049194 but that just gave me these: caller=main.go:740 err="opening storage failed: repair corrupted WAL: cannot handle error: open WAL segment: 0: open /prometheus/wal/00000000: no such file or directory" Not sure what I'm supposed to do with that. I just fails every time I try to start it. The documentation for 1.8 recommends: If you suspect problems caused by corruption in the database, you can enforce a crash recovery by starting the server with the flag storage.local.dirty. https://prometheus.io/docs/prometheus/1.8/storage/#data-corruption But when I tried using --storage.local.dirty I got back: Error parsing commandline arguments: unknown long flag '--storage.local.dirty' So that option is no longer there, but I can't find an alternative here: https://prometheus.io/docs/prometheus/2.16/storage/ It states that: If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory. Non POSIX compliant filesystems are not supported by Prometheus's local storage, corruptions may happen, without possibility to recover. And that: You can try removing individual block directories to resolve the problem, this means losing a time window of around two hours worth of data per block directory. But /prometheus/wal/00000000 doesn't exist, so I'm not sure what I'm supposed to remove. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/74d940f9-8aac-4e09-9794-d739487851f5%40googlegroups.com.
