We are using prometheus version 2.15.2 and started seeing compaction failures that are causing all database files after the initial issue to fail to load.
In the graph this looks like a large gap from the start of failure to where the in-memory database begins. I've ******** out exact names, in console directory names and timestamps are displayed. level=info ts=2020-04-02T19:00:06.111Z caller=compact.go:496 component=tsdb msg="write block" mint=******** maxt=******** ulid=************ duration= 5.648508245s level=info ts=2020-04-02T19:00:07.180Z caller=head.go:668 component=tsdb msg ="head GC completed" duration=115.070561ms level=error ts=2020-04-02T19:00:13.304Z caller=db.go:609 component=tsdb msg="compaction failed" err="delete persisted head block after failed db reload:*****************: unlinkat /prometheus/****************/chunks: directory not empty" level=error ts=2020-04-02T19:00:14.572Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory" level=error ts=2020-04-02T19:01:18.920Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory" level=error ts=2020-04-02T19:02:27.367Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory" level=error ts=2020-04-02T19:03:45.459Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory" Is anyone familiar with this error? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b087ed1d-33fe-4245-b3f5-2f356c8a3d0a%40googlegroups.com.
