Hello , We are facing an issue with Prometheus which is running on Azure Kubernetes Service (v1.19.1). Nodes are Ubuntu 18.04. We are using Azure Storage File Share to store the prometheus data. The Prometheus pods runs for couple of hours if we clear the data but then keeps going to CrashLoopBackOff and throws multiple errors. These are the Logs which we have gathered on multiple instances. We were using Prometheus v2.19.1 but we tried with Prometheus version 2.23.0 also but doesn't improve. The issue started since we upgraded the Kubernetes version from 1.17.9 to 1.19.1. Is there any compatibility issue during upgrade? ===============================================
level=info ts=2021-01-15T07:25:31.426Z caller=main.go:360 msg="Starting Prometheus" version="(version=2.23.0, branch=HEAD, revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" level=info ts=2021-01-15T07:25:31.426Z caller=main.go:365 build_context="(go=go1.15.5, user=root@37609b3a0a21, date=20201126-10:56:17)" level=info ts=2021-01-15T07:25:31.426Z caller=main.go:366 host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" level=info ts=2021-01-15T07:25:31.426Z caller=main.go:367 fd_limits="(soft=1048576, hard=1048576)" level=info ts=2021-01-15T07:25:31.426Z caller=main.go:368 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2021-01-15T07:25:31.526Z caller=main.go:722 msg="Starting TSDB ..." level=info ts=2021-01-15T07:25:31.526Z caller=web.go:528 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2021-01-15T07:25:31.757Z caller=head.go:645 component=tsdb msg="Replaying on-disk memory mappable chunks if any" level=info ts=2021-01-15T07:25:31.757Z caller=head.go:659 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.5µs level=info ts=2021-01-15T07:25:31.757Z caller=head.go:665 component=tsdb msg="Replaying WAL, this may take a while" level=info ts=2021-01-15T07:25:31.824Z caller=head.go:717 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2021-01-15T07:25:31.824Z caller=head.go:722 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.473299ms wal_replay_duration=43.082181ms total_replay_duration=66.57608ms level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 fs_type=fe534d42 level=info ts=2021-01-15T07:25:31.861Z caller=main.go:745 msg="TSDB started" level=info ts=2021-01-15T07:25:31.862Z caller=main.go:871 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2021-01-15T07:25:31.950Z caller=main.go:902 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=88.228971ms remote_storage=2.901µs web_handler=500ns query_engine=1.2µs scrape=380.501µs scrape_sd=135.301µs notify=27.1µs notify_sd=14.6µs rules=67.262483ms level=info ts=2021-01-15T07:25:31.950Z caller=main.go:694 msg="Server is ready to receive web requests." level=error ts=2021-01-15T07:52:33.026Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T08:18:36.113Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T08:21:38.206Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T08:24:40.278Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T08:40:42.841Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T09:50:47.814Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T10:26:03.298Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: seek /prometheus/01EW2S98EFR81VFT0677NH81ZF.tmp-for-creation/chunks/000001.tmp: interrupted system call" level=error ts=2021-01-15T10:26:05.816Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: seek /prometheus/01EW2S9AXV1E42NHPGV0K9Z2MC.tmp-for-creation/chunks/000001.tmp: interrupted system call" level=error ts=2021-01-15T10:26:15.075Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: seek /prometheus/01EW2S9KZ7GZZ1N36NFMNDCKRV.tmp-for-creation/chunks/000001.tmp: interrupted system call" level=info ts=2021-01-15T10:26:21.209Z caller=compact.go:502 component=tsdb msg="write block" mint=1610695556030 maxt=1610697600000 ulid=01EW2S9SMSDCR35G93DT27G6NR duration=767.820675ms level=info ts=2021-01-15T10:26:21.412Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=25.718613ms level=error ts=2021-01-15T10:36:22.279Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T10:55:25.565Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T10:56:27.577Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=info ts=2021-01-15T11:00:06.243Z caller=compact.go:502 component=tsdb msg="write block" mint=1610697600000 maxt=1610704800000 ulid=01EW2V7K5TDBRGNWYZXMMN50C7 duration=808.923904ms level=info ts=2021-01-15T11:00:06.487Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=33.110139ms level=error ts=2021-01-15T11:48:11.574Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T11:55:14.296Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T12:09:17.495Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T12:24:20.974Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T12:51:25.739Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T12:55:28.056Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T13:00:02.893Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: open chunk writer: mkdir /prometheus/01EW3237WF9WWH4TDY3HA2KBRK.tmp-for-creation/chunks: interrupted system call" level=error ts=2021-01-15T13:00:05.471Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: open chunk writer: mkdir /prometheus/01EW323AD1YAQ6VXB2G6Q6HZ58.tmp-for-creation/chunks: interrupted system call" level=info ts=2021-01-15T13:00:15.444Z caller=compact.go:502 component=tsdb msg="write block" mint=1610704800000 maxt=1610712000000 ulid=01EW323KD7FYEMJ0V03RX7TDR7 duration=812.389067ms level=info ts=2021-01-15T13:00:15.725Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=31.144829ms level=error ts=2021-01-15T13:07:16.892Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T13:35:22.717Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T13:42:25.587Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T13:47:28.162Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T13:55:31.084Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T14:19:36.277Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T14:24:38.878Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T14:54:45.004Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T14:56:47.144Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T15:00:06.054Z caller=compact.go:541 component=tsdb msg="removed tmp folder after failed compaction" err="unlinkat /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation: directory not empty" level=error ts=2021-01-15T15:00:06.054Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: sync /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation/index: interrupted system call" level=info ts=2021-01-15T15:00:07.791Z caller=compact.go:502 component=tsdb msg="write block" mint=1610712000000 maxt=1610719200000 ulid=01EW38Z38E2TA8CDEJNJ130QV9 duration=736.679699ms level=info ts=2021-01-15T15:00:08.081Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=27.759017ms level=info ts=2021-01-15T15:00:08.261Z caller=checkpoint.go:95 component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 mint=1610719200000 level=info ts=2021-01-15T15:00:08.689Z caller=head.go:908 component=tsdb msg="WAL checkpoint complete" first=0 last=1 duration=454.463412ms level=error ts=2021-01-15T15:01:08.914Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T15:17:13.515Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T15:44:19.858Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T15:50:22.808Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T15:54:25.360Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T15:55:27.371Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T16:02:32.493Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T16:09:35.501Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T16:17:38.646Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T16:19:40.796Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T16:28:44.071Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=info ts=2021-01-15T17:00:01.827Z caller=compact.go:502 component=tsdb msg="write block" mint=1610719200000 maxt=1610726400000 ulid=01EW3FTMKQYF12V439SBT6B17P duration=811.71891ms level=info ts=2021-01-15T17:00:02.175Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=28.805321ms level=error ts=2021-01-15T17:00:03.791Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact [/prometheus/01EW2S9SMSDCR35G93DT27G6NR /prometheus/01EW2V7K5TDBRGNWYZXMMN50C7 /prometheus/01EW323KD7FYEMJ0V03RX7TDR7]: write merged meta: rename /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json.tmp /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json: interrupted system call" level=error ts=2021-01-15T17:01:05.830Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T17:03:10.067Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T17:18:14.829Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T17:24:17.637Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T17:36:21.558Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T17:38:23.743Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T17:48:27.143Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T18:12:32.875Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T18:20:36.002Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T18:24:38.461Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T18:32:41.690Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T18:36:44.188Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T18:42:47.011Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T19:00:06.007Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: write merged meta: rename /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json.tmp /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json: interrupted system call" level=info ts=2021-01-15T19:00:07.649Z caller=compact.go:502 component=tsdb msg="write block" mint=1610726400000 maxt=1610733600000 ulid=01EW3PPHPZ6ZZM2JGKF69F96RN duration=641.827141ms level=info ts=2021-01-15T19:00:07.932Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=24.2159ms level=info ts=2021-01-15T19:00:08.096Z caller=checkpoint.go:95 component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 mint=1610733600000 level=info ts=2021-01-15T19:00:08.589Z caller=head.go:908 component=tsdb msg="WAL checkpoint complete" first=2 last=3 duration=516.496925ms level=error ts=2021-01-15T19:08:09.957Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T19:16:13.284Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T19:19:15.693Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T19:20:17.699Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T19:39:24.830Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T19:42:27.215Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T19:45:29.593Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T19:56:33.420Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:06:37.228Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:08:39.420Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:19:43.242Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:30:47.040Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:35:49.798Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:38:52.166Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:49:55.966Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:50:57.973Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T20:59:03.232Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T21:00:05.262Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T21:00:07.878Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: write merged meta: rename /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json.tmp /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json: interrupted system call" level=info ts=2021-01-15T21:00:12.523Z caller=compact.go:502 component=tsdb msg="write block" mint=1610733600000 maxt=1610740800000 ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR duration=644.683302ms level=info ts=2021-01-15T21:00:12.828Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=22.771295ms level=info ts=2021-01-15T21:00:13.960Z caller=compact.go:443 component=tsdb msg="compact blocks" count=3 mint=1610712000000 maxt=1610733600000 ulid=01EW3XJEZR86TBBDJFVTBNZ8VF sources="[01EW38Z38E2TA8CDEJNJ130QV9 01EW3FTMKQYF12V439SBT6B17P 01EW3PPHPZ6ZZM2JGKF69F96RN]" duration=827.624569ms level=info ts=2021-01-15T21:00:14.381Z caller=db.go:1187 component=tsdb msg="Deleting obsolete block" block=01EW38Z38E2TA8CDEJNJ130QV9 level=info ts=2021-01-15T21:00:14.547Z caller=db.go:1187 component=tsdb msg="Deleting obsolete block" block=01EW3FTMKQYF12V439SBT6B17P level=info ts=2021-01-15T21:00:14.705Z caller=db.go:1187 component=tsdb msg="Deleting obsolete block" block=01EW3PPHPZ6ZZM2JGKF69F96RN level=error ts=2021-01-15T21:03:15.152Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T21:20:19.710Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T21:36:24.045Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T21:38:26.227Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T21:46:29.339Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T21:47:31.378Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T22:00:37.302Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T22:02:39.487Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T22:17:43.600Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T22:26:47.082Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T22:35:50.353Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T22:36:52.383Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T23:00:06.094Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: write new tombstones file: rename /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones.tmp /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones: interrupted system call" level=error ts=2021-01-15T23:00:07.633Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: write merged meta: rename /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json.tmp /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json: interrupted system call" level=error ts=2021-01-15T23:00:13.695Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: rename block dir: rename /prometheus/01EW44E60NC3EGCETC25GBW492.tmp-for-creation /prometheus/01EW44E60NC3EGCETC25GBW492: interrupted system call" level=error ts=2021-01-15T23:00:18.059Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: seek /prometheus/01EW44EAMZKJ5JJ2E42WCM8V0Z.tmp-for-creation/chunks/000001.tmp: interrupted system call" level=error ts=2021-01-15T23:00:26.689Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: write merged meta: rename /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json.tmp /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json: interrupted system call" level=error ts=2021-01-15T23:00:43.395Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: write new tombstones file: rename /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones.tmp /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones: interrupted system call" level=info ts=2021-01-15T23:01:16.055Z caller=compact.go:502 component=tsdb msg="write block" mint=1610740800000 maxt=1610748000000 ulid=01EW44G303K0DNBB2EDM8DX69E duration=659.365701ms level=info ts=2021-01-15T23:01:16.353Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=23.148498ms level=info ts=2021-01-15T23:01:16.514Z caller=checkpoint.go:95 component=tsdb msg="Creating checkpoint" from_segment=4 to_segment=5 mint=1610748000000 level=info ts=2021-01-15T23:01:17.004Z caller=head.go:908 component=tsdb msg="WAL checkpoint complete" first=4 last=5 duration=512.755478ms level=error ts=2021-01-15T23:22:20.652Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T23:25:23.109Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T23:40:27.567Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-15T23:50:31.230Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T00:03:35.502Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T00:20:40.429Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T00:21:42.458Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T00:23:46.659Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T00:25:48.862Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T00:37:52.910Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T00:57:58.400Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T01:00:00.602Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=info ts=2021-01-16T01:00:06.068Z caller=compact.go:502 component=tsdb msg="write block" mint=1610748000000 maxt=1610755200000 ulid=01EW4B9NW6T794HCZATQJZJ4SZ duration=686.024281ms level=info ts=2021-01-16T01:00:06.429Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=22.475094ms level=error ts=2021-01-16T01:00:06.563Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: remove /prometheus/chunks_head/000007: interrupted system call" level=error ts=2021-01-16T01:08:12.075Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T01:15:15.300Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T01:20:18.133Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T01:22:20.357Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T01:31:24.051Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T01:50:29.872Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T01:52:32.155Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T02:00:35.607Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T02:40:45.873Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T02:54:50.472Z caller=db.go:746 component=tsdb msg="compaction failed" err="plan compaction: readdirent: no such file or directory" level=error ts=2021-01-16T03:00:06.014Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: write merged meta: rename /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json.tmp /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json: interrupted system call" level=error ts=2021-01-16T03:00:16.046Z caller=compact.go:541 component=tsdb msg="removed tmp folder after failed compaction" err="unlinkat /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation: directory not empty" level=error ts=2021-01-16T03:00:16.046Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: sync /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation/index: interrupted system call" level=error ts=2021-01-16T03:00:20.488Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: open chunk writer: mkdir /prometheus/01EW4J5VTSRT3KXABDFXETHK99.tmp-for-creation/chunks: interrupted system call" level=error ts=2021-01-16T03:00:25.202Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact head: persist head block: write new tombstones file: rename /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones.tmp /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones: interrupted system call" level=info ts=2021-01-16T03:00:33.853Z caller=compact.go:502 component=tsdb msg="write block" mint=1610755200000 maxt=1610762400000 ulid=01EW4J689JDY2GWPCPDNHD0DNJ duration=650.542029ms level=info ts=2021-01-16T03:00:34.211Z caller=head.go:824 component=tsdb msg="Head GC completed" duration=22.278794ms level=error ts=2021-01-16T03:00:35.654Z caller=db.go:746 component=tsdb msg="compaction failed" err="compact [/prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR /prometheus/01EW44G303K0DNBB2EDM8DX69E /prometheus/01EW4B9NW6T794HCZATQJZJ4SZ]: 2 errors: rename block dir: rename /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK.tmp-for-creation /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK: interrupted system call; setting compaction failed for block: /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR: rename /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json.tmp /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json: interrupted system call" =============================== level=info ts=2021-01-19T05:06:08.707Z caller=main.go:360 msg="Starting Prometheus" version="(version=2.23.0, branch=HEAD, revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" level=info ts=2021-01-19T05:06:08.707Z caller=main.go:365 build_context="(go=go1.15.5, user=root@37609b3a0a21, date=20201126-10:56:17)" level=info ts=2021-01-19T05:06:08.707Z caller=main.go:366 host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" level=info ts=2021-01-19T05:06:08.707Z caller=main.go:367 fd_limits="(soft=1048576, hard=1048576)" level=info ts=2021-01-19T05:06:08.707Z caller=main.go:368 vm_limits="(soft=unlimited, hard=unlimited)" level=error ts=2021-01-19T05:06:08.726Z caller=query_logger.go:71 component=activeQueryTracker msg="Failed to read query log file" err=EOF unexpected fault address 0x7f3c06738000 fatal error: fault [signal SIGBUS: bus error code=0x2 addr=0x7f3c06738000 pc=0x470ea8] goroutine 1 [running]: runtime.throw(0x29f4a25, 0x5) /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00092b248 sp=0xc00092b218 pc=0x4378b2 runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:739 +0x485 fp=0xc00092b278 sp=0xc00092b248 pc=0x44e085 runtime.memmove(0x7f3c06738000, 0x29f1e48, 0x1) /usr/local/go/src/runtime/memmove_amd64.s:148 +0x108 fp=0xc00092b280 sp=0xc00092b278 pc=0x470ea8 github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff842a774d, 0xb, 0x14, 0x30d9a40, 0xc00077ede0, 0x30d9a40) /app/promql/query_logger.go:120 +0x3a5 fp=0xc00092b3d8 sp=0xc00092b280 pc=0x20fac05 main.main() /app/cmd/prometheus/main.go:396 +0x53cc fp=0xc00092bf88 sp=0xc00092b3d8 pc=0x21f9a0c runtime.main() /usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc00092bfe0 sp=0xc00092bf88 pc=0x43a0a9 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00092bfe8 sp=0xc00092bfe0 pc=0x46ffa1 goroutine 9 [select]: go.opencensus.io/stats/view.(*worker).start(0xc00030c280) /app/vendor/go.opencensus.io/stats/view/worker.go:276 +0x105 created by go.opencensus.io/stats/view.init.0 /app/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68 goroutine 85 [select]: github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc000276780) /app/pkg/logging/dedupe.go:75 +0x1e5 created by github.com/prometheus/prometheus/pkg/logging.Dedupe /app/pkg/logging/dedupe.go:61 +0xcf goroutine 92 [chan receive]: github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc00018ad20) /app/storage/remote/write.go:96 +0xb6 created by github.com/prometheus/prometheus/storage/remote.NewWriteStorage /app/storage/remote/write.go:89 +0x305 ============================================================================ level=info ts=2021-01-19T05:42:16.710Z caller=main.go:360 msg="Starting Prometheus" version="(version=2.23.0, branch=HEAD, revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" level=info ts=2021-01-19T05:42:16.711Z caller=main.go:365 build_context="(go=go1.15.5, user=root@37609b3a0a21, date=20201126-10:56:17)" level=info ts=2021-01-19T05:42:16.711Z caller=main.go:366 host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" level=info ts=2021-01-19T05:42:16.711Z caller=main.go:367 fd_limits="(soft=1048576, hard=1048576)" level=info ts=2021-01-19T05:42:16.711Z caller=main.go:368 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2021-01-19T05:42:16.796Z caller=web.go:528 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2021-01-19T05:42:16.796Z caller=main.go:722 msg="Starting TSDB ..." level=info ts=2021-01-19T05:42:16.844Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610695556030 maxt=1610697600000 ulid=01EW2S9SMSDCR35G93DT27G6NR level=info ts=2021-01-19T05:42:16.875Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610697600000 maxt=1610704800000 ulid=01EW2V7K5TDBRGNWYZXMMN50C7 level=info ts=2021-01-19T05:42:16.901Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610704800000 maxt=1610712000000 ulid=01EW323KD7FYEMJ0V03RX7TDR7 level=info ts=2021-01-19T05:42:16.925Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610733600000 maxt=1610740800000 ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR level=info ts=2021-01-19T05:42:16.951Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610712000000 maxt=1610733600000 ulid=01EW3XJEZR86TBBDJFVTBNZ8VF level=info ts=2021-01-19T05:42:16.976Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610740800000 maxt=1610748000000 ulid=01EW44G303K0DNBB2EDM8DX69E level=info ts=2021-01-19T05:42:17.001Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610748000000 maxt=1610755200000 ulid=01EW4B9NW6T794HCZATQJZJ4SZ level=info ts=2021-01-19T05:42:17.049Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610755200000 maxt=1610762400000 ulid=01EW4J689JDY2GWPCPDNHD0DNJ level=info ts=2021-01-19T05:42:17.074Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610762400000 maxt=1610769600000 ulid=01EW4S14CHH73BZHP8152M3594 level=info ts=2021-01-19T05:42:17.099Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610769600000 maxt=1610776800000 ulid=01EW4ZXAASHGVGEY6QZ4C6R47M level=info ts=2021-01-19T05:42:17.125Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610798400000 maxt=1610805600000 ulid=01EW5VBMBQBYV98Q278AHHNK70 level=info ts=2021-01-19T05:42:17.155Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610776800000 maxt=1610798400000 ulid=01EW5VBP3B7KSWNETV0AEKFSWY level=info ts=2021-01-19T05:42:17.184Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610805600000 maxt=1610812800000 ulid=01EW627M0DHYWTCC0N237SC62D level=info ts=2021-01-19T05:42:17.234Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610812800000 maxt=1610820000000 ulid=01EW6939M4CNXWMHGT34SD5N6R level=info ts=2021-01-19T05:42:17.259Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610820000000 maxt=1610841600000 ulid=01EW74JPT1T99G30CWTAE8YTVC level=info ts=2021-01-19T05:42:17.285Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610863200000 maxt=1610870400000 ulid=01EW7S59W18PRNC14KHMY2NQDF level=info ts=2021-01-19T05:42:17.311Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610841600000 maxt=1610863200000 ulid=01EW7S5CEZAKV5FHJD1BYN7VFS level=info ts=2021-01-19T05:42:17.337Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610870400000 maxt=1610877600000 ulid=01EW8012X70MJZYVQTEKERJSF0 level=info ts=2021-01-19T05:42:17.393Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610877600000 maxt=1610884800000 ulid=01EW86X36BN7KDN3YQY4KH1X7Z level=info ts=2021-01-19T05:42:17.418Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610884800000 maxt=1610892000000 ulid=01EW8DRFQEXQP7TCA6W9ZCWJJC level=info ts=2021-01-19T05:42:17.443Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1610863200000 maxt=1610884800000 ulid=01EW8DRHV7JX5CV9YKDFYBQQ99 level=info ts=2021-01-19T05:42:17.606Z caller=main.go:595 msg="Stopping scrape discovery manager..." level=info ts=2021-01-19T05:42:17.607Z caller=main.go:609 msg="Stopping notify discovery manager..." level=info ts=2021-01-19T05:42:17.607Z caller=main.go:631 msg="Stopping scrape manager..." level=info ts=2021-01-19T05:42:17.607Z caller=main.go:605 msg="Notify discovery manager stopped" level=info ts=2021-01-19T05:42:17.607Z caller=main.go:591 msg="Scrape discovery manager stopped" level=info ts=2021-01-19T05:42:17.607Z caller=manager.go:924 component="rule manager" msg="Stopping rule manager..." level=info ts=2021-01-19T05:42:17.608Z caller=manager.go:934 component="rule manager" msg="Rule manager stopped" level=info ts=2021-01-19T05:42:17.608Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." level=info ts=2021-01-19T05:42:17.608Z caller=main.go:799 msg="Notifier manager stopped" level=info ts=2021-01-19T05:42:17.608Z caller=main.go:625 msg="Scrape manager stopped" level=error ts=2021-01-19T05:42:17.608Z caller=main.go:808 err="opening storage failed: found unsequential head chunk files /prometheus/chunks_head/000007 (index: 7) and /prometheus/chunks_head/000027 (index: 27)" ======================================= level=info ts=2021-01-19T10:55:57.643Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.19.1, branch=HEAD, revision=eba3fdcbf0d378b66600281903e3aab515732b39)" level=info ts=2021-01-19T10:55:57.644Z caller=main.go:338 build_context="(go=go1.14.4, user=root@62700b3d0ef9, date=20200618-16:35:26)" level=info ts=2021-01-19T10:55:57.644Z caller=main.go:339 host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 09:13:52 UTC 2020 x86_64 prometheus-84fb48b4d9-ddq9m (none))" level=info ts=2021-01-19T10:55:57.644Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" level=info ts=2021-01-19T10:55:57.644Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" level=error ts=2021-01-19T10:55:57.704Z caller=query_logger.go:93 component=activeQueryTracker msg="Error setting filesize." filesize=20001 err="truncate /prometheus/queries.active: interrupted system call" panic: Unable to create mmap-ed active query log goroutine 1 [running]: github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff2f2aa74d, 0xb, 0x14, 0x2f3b7e0, 0xc00074d800, 0x2f3b7e0) /app/promql/query_logger.go:117 +0x4cd main.main() /app/cmd/prometheus/main.go:368 +0x4ed2 Appreciate any help Thanks. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/43ecf3a9-005f-4b2a-a16d-fa6cf065916an%40googlegroups.com.
