We are not attaching Prometheus directly with NFS or SMB. We are using PVC, there Kubernetes connect using protocol. Container only see as regular volume attached as it is, it is transparent with Kubernetes as per my understanding. On Tuesday, January 19, 2021 at 5:39:43 PM UTC+5:30 Sdh26 wrote:
> Hello , > > We are facing an issue with Prometheus which is running on Azure > Kubernetes Service (v1.19.1). Nodes are Ubuntu 18.04. > We are using Azure Storage File Share to store the prometheus data. > The Prometheus pods runs for couple of hours if we clear the data but then > keeps going to CrashLoopBackOff and throws multiple errors. > These are the Logs which we have gathered on multiple instances. We were > using Prometheus v2.19.1 but we tried with Prometheus version 2.23.0 also > but doesn't improve. > The issue started since we upgraded the Kubernetes version from 1.17.9 to > 1.19.1. > Is there any compatibility issue during upgrade? > =============================================== > > level=info ts=2021-01-15T07:25:31.426Z caller=main.go:360 msg="Starting > Prometheus" version="(version=2.23.0, branch=HEAD, > revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > level=info ts=2021-01-15T07:25:31.426Z caller=main.go:365 > build_context="(go=go1.15.5, user=root@37609b3a0a21, > date=20201126-10:56:17)" > level=info ts=2021-01-15T07:25:31.426Z caller=main.go:366 > host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 > 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > level=info ts=2021-01-15T07:25:31.426Z caller=main.go:367 > fd_limits="(soft=1048576, hard=1048576)" > level=info ts=2021-01-15T07:25:31.426Z caller=main.go:368 > vm_limits="(soft=unlimited, hard=unlimited)" > level=info ts=2021-01-15T07:25:31.526Z caller=main.go:722 msg="Starting > TSDB ..." > level=info ts=2021-01-15T07:25:31.526Z caller=web.go:528 component=web > msg="Start listening for connections" address=0.0.0.0:9090 > level=info ts=2021-01-15T07:25:31.757Z caller=head.go:645 component=tsdb > msg="Replaying on-disk memory mappable chunks if any" > level=info ts=2021-01-15T07:25:31.757Z caller=head.go:659 component=tsdb > msg="On-disk memory mappable chunks replay completed" duration=4.5µs > level=info ts=2021-01-15T07:25:31.757Z caller=head.go:665 component=tsdb > msg="Replaying WAL, this may take a while" > level=info ts=2021-01-15T07:25:31.824Z caller=head.go:717 component=tsdb > msg="WAL segment loaded" segment=0 maxSegment=0 > level=info ts=2021-01-15T07:25:31.824Z caller=head.go:722 component=tsdb > msg="WAL replay completed" checkpoint_replay_duration=23.473299ms > wal_replay_duration=43.082181ms total_replay_duration=66.57608ms > level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 fs_type=fe534d42 > level=info ts=2021-01-15T07:25:31.861Z caller=main.go:745 msg="TSDB > started" > level=info ts=2021-01-15T07:25:31.862Z caller=main.go:871 msg="Loading > configuration file" filename=/etc/prometheus/prometheus.yml > level=info ts=2021-01-15T07:25:31.950Z caller=main.go:902 msg="Completed > loading of configuration file" filename=/etc/prometheus/prometheus.yml > totalDuration=88.228971ms remote_storage=2.901µs web_handler=500ns > query_engine=1.2µs scrape=380.501µs scrape_sd=135.301µs notify=27.1µs > notify_sd=14.6µs rules=67.262483ms > level=info ts=2021-01-15T07:25:31.950Z caller=main.go:694 msg="Server is > ready to receive web requests." > level=error ts=2021-01-15T07:52:33.026Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T08:18:36.113Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T08:21:38.206Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T08:24:40.278Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T08:40:42.841Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T09:50:47.814Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T10:26:03.298Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: populate > block: write chunks: preallocate: seek > /prometheus/01EW2S98EFR81VFT0677NH81ZF.tmp-for-creation/chunks/000001.tmp: > interrupted system call" > level=error ts=2021-01-15T10:26:05.816Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: populate > block: write chunks: preallocate: seek > /prometheus/01EW2S9AXV1E42NHPGV0K9Z2MC.tmp-for-creation/chunks/000001.tmp: > interrupted system call" > level=error ts=2021-01-15T10:26:15.075Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: populate > block: write chunks: preallocate: seek > /prometheus/01EW2S9KZ7GZZ1N36NFMNDCKRV.tmp-for-creation/chunks/000001.tmp: > interrupted system call" > level=info ts=2021-01-15T10:26:21.209Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610695556030 maxt=1610697600000 > ulid=01EW2S9SMSDCR35G93DT27G6NR duration=767.820675ms > level=info ts=2021-01-15T10:26:21.412Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=25.718613ms > level=error ts=2021-01-15T10:36:22.279Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T10:55:25.565Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T10:56:27.577Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=info ts=2021-01-15T11:00:06.243Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610697600000 maxt=1610704800000 > ulid=01EW2V7K5TDBRGNWYZXMMN50C7 duration=808.923904ms > level=info ts=2021-01-15T11:00:06.487Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=33.110139ms > level=error ts=2021-01-15T11:48:11.574Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T11:55:14.296Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T12:09:17.495Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T12:24:20.974Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T12:51:25.739Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T12:55:28.056Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T13:00:02.893Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: open chunk > writer: mkdir > /prometheus/01EW3237WF9WWH4TDY3HA2KBRK.tmp-for-creation/chunks: interrupted > system call" > level=error ts=2021-01-15T13:00:05.471Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: open chunk > writer: mkdir > /prometheus/01EW323AD1YAQ6VXB2G6Q6HZ58.tmp-for-creation/chunks: interrupted > system call" > level=info ts=2021-01-15T13:00:15.444Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610704800000 maxt=1610712000000 > ulid=01EW323KD7FYEMJ0V03RX7TDR7 duration=812.389067ms > level=info ts=2021-01-15T13:00:15.725Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=31.144829ms > level=error ts=2021-01-15T13:07:16.892Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T13:35:22.717Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T13:42:25.587Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T13:47:28.162Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T13:55:31.084Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T14:19:36.277Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T14:24:38.878Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T14:54:45.004Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T14:56:47.144Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T15:00:06.054Z caller=compact.go:541 > component=tsdb msg="removed tmp folder after failed compaction" > err="unlinkat /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation: > directory not empty" > level=error ts=2021-01-15T15:00:06.054Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: sync > /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation/index: interrupted > system call" > level=info ts=2021-01-15T15:00:07.791Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610712000000 maxt=1610719200000 > ulid=01EW38Z38E2TA8CDEJNJ130QV9 duration=736.679699ms > level=info ts=2021-01-15T15:00:08.081Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=27.759017ms > level=info ts=2021-01-15T15:00:08.261Z caller=checkpoint.go:95 > component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 > mint=1610719200000 > level=info ts=2021-01-15T15:00:08.689Z caller=head.go:908 component=tsdb > msg="WAL checkpoint complete" first=0 last=1 duration=454.463412ms > level=error ts=2021-01-15T15:01:08.914Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T15:17:13.515Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T15:44:19.858Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T15:50:22.808Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T15:54:25.360Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T15:55:27.371Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T16:02:32.493Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T16:09:35.501Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T16:17:38.646Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T16:19:40.796Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T16:28:44.071Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=info ts=2021-01-15T17:00:01.827Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610719200000 maxt=1610726400000 > ulid=01EW3FTMKQYF12V439SBT6B17P duration=811.71891ms > level=info ts=2021-01-15T17:00:02.175Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=28.805321ms > level=error ts=2021-01-15T17:00:03.791Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact > [/prometheus/01EW2S9SMSDCR35G93DT27G6NR > /prometheus/01EW2V7K5TDBRGNWYZXMMN50C7 > /prometheus/01EW323KD7FYEMJ0V03RX7TDR7]: write merged meta: rename > /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json.tmp > /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json: > interrupted system call" > level=error ts=2021-01-15T17:01:05.830Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T17:03:10.067Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T17:18:14.829Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T17:24:17.637Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T17:36:21.558Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T17:38:23.743Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T17:48:27.143Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T18:12:32.875Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T18:20:36.002Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T18:24:38.461Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T18:32:41.690Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T18:36:44.188Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T18:42:47.011Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T19:00:06.007Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: write merged > meta: rename > /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json.tmp > /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json: > interrupted system call" > level=info ts=2021-01-15T19:00:07.649Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610726400000 maxt=1610733600000 > ulid=01EW3PPHPZ6ZZM2JGKF69F96RN duration=641.827141ms > level=info ts=2021-01-15T19:00:07.932Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=24.2159ms > level=info ts=2021-01-15T19:00:08.096Z caller=checkpoint.go:95 > component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 > mint=1610733600000 > level=info ts=2021-01-15T19:00:08.589Z caller=head.go:908 component=tsdb > msg="WAL checkpoint complete" first=2 last=3 duration=516.496925ms > level=error ts=2021-01-15T19:08:09.957Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T19:16:13.284Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T19:19:15.693Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T19:20:17.699Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T19:39:24.830Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T19:42:27.215Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T19:45:29.593Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T19:56:33.420Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:06:37.228Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:08:39.420Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:19:43.242Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:30:47.040Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:35:49.798Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:38:52.166Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:49:55.966Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:50:57.973Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T20:59:03.232Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T21:00:05.262Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T21:00:07.878Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: write merged > meta: rename > /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json.tmp > /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json: > interrupted system call" > level=info ts=2021-01-15T21:00:12.523Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610733600000 maxt=1610740800000 > ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR duration=644.683302ms > level=info ts=2021-01-15T21:00:12.828Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=22.771295ms > level=info ts=2021-01-15T21:00:13.960Z caller=compact.go:443 > component=tsdb msg="compact blocks" count=3 mint=1610712000000 > maxt=1610733600000 ulid=01EW3XJEZR86TBBDJFVTBNZ8VF > sources="[01EW38Z38E2TA8CDEJNJ130QV9 01EW3FTMKQYF12V439SBT6B17P > 01EW3PPHPZ6ZZM2JGKF69F96RN]" duration=827.624569ms > level=info ts=2021-01-15T21:00:14.381Z caller=db.go:1187 component=tsdb > msg="Deleting obsolete block" block=01EW38Z38E2TA8CDEJNJ130QV9 > level=info ts=2021-01-15T21:00:14.547Z caller=db.go:1187 component=tsdb > msg="Deleting obsolete block" block=01EW3FTMKQYF12V439SBT6B17P > level=info ts=2021-01-15T21:00:14.705Z caller=db.go:1187 component=tsdb > msg="Deleting obsolete block" block=01EW3PPHPZ6ZZM2JGKF69F96RN > level=error ts=2021-01-15T21:03:15.152Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T21:20:19.710Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T21:36:24.045Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T21:38:26.227Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T21:46:29.339Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T21:47:31.378Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T22:00:37.302Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T22:02:39.487Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T22:17:43.600Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T22:26:47.082Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T22:35:50.353Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T22:36:52.383Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T23:00:06.094Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: write new > tombstones file: rename > /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones.tmp > /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones: > interrupted system call" > level=error ts=2021-01-15T23:00:07.633Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: write merged > meta: rename > /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json.tmp > /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json: > interrupted system call" > level=error ts=2021-01-15T23:00:13.695Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: rename block > dir: rename /prometheus/01EW44E60NC3EGCETC25GBW492.tmp-for-creation > /prometheus/01EW44E60NC3EGCETC25GBW492: interrupted system call" > level=error ts=2021-01-15T23:00:18.059Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: populate > block: write chunks: preallocate: seek > /prometheus/01EW44EAMZKJ5JJ2E42WCM8V0Z.tmp-for-creation/chunks/000001.tmp: > interrupted system call" > level=error ts=2021-01-15T23:00:26.689Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: write merged > meta: rename > /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json.tmp > /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json: > interrupted system call" > level=error ts=2021-01-15T23:00:43.395Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: write new > tombstones file: rename > /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones.tmp > /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones: > interrupted system call" > level=info ts=2021-01-15T23:01:16.055Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610740800000 maxt=1610748000000 > ulid=01EW44G303K0DNBB2EDM8DX69E duration=659.365701ms > level=info ts=2021-01-15T23:01:16.353Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=23.148498ms > level=info ts=2021-01-15T23:01:16.514Z caller=checkpoint.go:95 > component=tsdb msg="Creating checkpoint" from_segment=4 to_segment=5 > mint=1610748000000 > level=info ts=2021-01-15T23:01:17.004Z caller=head.go:908 component=tsdb > msg="WAL checkpoint complete" first=4 last=5 duration=512.755478ms > level=error ts=2021-01-15T23:22:20.652Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T23:25:23.109Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T23:40:27.567Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-15T23:50:31.230Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T00:03:35.502Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T00:20:40.429Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T00:21:42.458Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T00:23:46.659Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T00:25:48.862Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T00:37:52.910Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T00:57:58.400Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T01:00:00.602Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=info ts=2021-01-16T01:00:06.068Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610748000000 maxt=1610755200000 > ulid=01EW4B9NW6T794HCZATQJZJ4SZ duration=686.024281ms > level=info ts=2021-01-16T01:00:06.429Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=22.475094ms > level=error ts=2021-01-16T01:00:06.563Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: head memory truncate: truncate > chunks.HeadReadWriter: remove /prometheus/chunks_head/000007: interrupted > system call" > level=error ts=2021-01-16T01:08:12.075Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T01:15:15.300Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T01:20:18.133Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T01:22:20.357Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T01:31:24.051Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T01:50:29.872Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T01:52:32.155Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T02:00:35.607Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T02:40:45.873Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T02:54:50.472Z caller=db.go:746 component=tsdb > msg="compaction failed" err="plan compaction: readdirent: no such file or > directory" > level=error ts=2021-01-16T03:00:06.014Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: write merged > meta: rename > /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json.tmp > /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json: > interrupted system call" > level=error ts=2021-01-16T03:00:16.046Z caller=compact.go:541 > component=tsdb msg="removed tmp folder after failed compaction" > err="unlinkat /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation: > directory not empty" > level=error ts=2021-01-16T03:00:16.046Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: sync > /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation/index: interrupted > system call" > level=error ts=2021-01-16T03:00:20.488Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: open chunk > writer: mkdir > /prometheus/01EW4J5VTSRT3KXABDFXETHK99.tmp-for-creation/chunks: interrupted > system call" > level=error ts=2021-01-16T03:00:25.202Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact head: persist head block: write new > tombstones file: rename > /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones.tmp > /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones: > interrupted system call" > level=info ts=2021-01-16T03:00:33.853Z caller=compact.go:502 > component=tsdb msg="write block" mint=1610755200000 maxt=1610762400000 > ulid=01EW4J689JDY2GWPCPDNHD0DNJ duration=650.542029ms > level=info ts=2021-01-16T03:00:34.211Z caller=head.go:824 component=tsdb > msg="Head GC completed" duration=22.278794ms > level=error ts=2021-01-16T03:00:35.654Z caller=db.go:746 component=tsdb > msg="compaction failed" err="compact > [/prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR > /prometheus/01EW44G303K0DNBB2EDM8DX69E > /prometheus/01EW4B9NW6T794HCZATQJZJ4SZ]: 2 errors: rename block dir: rename > /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK.tmp-for-creation > /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK: interrupted system call; setting > compaction failed for block: /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR: rename > /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json.tmp > /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json: interrupted system call" > > > =============================== > level=info ts=2021-01-19T05:06:08.707Z caller=main.go:360 msg="Starting > Prometheus" version="(version=2.23.0, branch=HEAD, > revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > level=info ts=2021-01-19T05:06:08.707Z caller=main.go:365 > build_context="(go=go1.15.5, user=root@37609b3a0a21, > date=20201126-10:56:17)" > level=info ts=2021-01-19T05:06:08.707Z caller=main.go:366 > host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 > 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > level=info ts=2021-01-19T05:06:08.707Z caller=main.go:367 > fd_limits="(soft=1048576, hard=1048576)" > level=info ts=2021-01-19T05:06:08.707Z caller=main.go:368 > vm_limits="(soft=unlimited, hard=unlimited)" > level=error ts=2021-01-19T05:06:08.726Z caller=query_logger.go:71 > component=activeQueryTracker msg="Failed to read query log file" err=EOF > unexpected fault address 0x7f3c06738000 > fatal error: fault > [signal SIGBUS: bus error code=0x2 addr=0x7f3c06738000 pc=0x470ea8] > > goroutine 1 [running]: > runtime.throw(0x29f4a25, 0x5) > /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00092b248 > sp=0xc00092b218 pc=0x4378b2 > runtime.sigpanic() > /usr/local/go/src/runtime/signal_unix.go:739 +0x485 fp=0xc00092b278 > sp=0xc00092b248 pc=0x44e085 > runtime.memmove(0x7f3c06738000, 0x29f1e48, 0x1) > /usr/local/go/src/runtime/memmove_amd64.s:148 +0x108 fp=0xc00092b280 > sp=0xc00092b278 pc=0x470ea8 > > github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff842a774d, > 0xb, 0x14, 0x30d9a40, 0xc00077ede0, 0x30d9a40) > /app/promql/query_logger.go:120 +0x3a5 fp=0xc00092b3d8 sp=0xc00092b280 > pc=0x20fac05 > main.main() > /app/cmd/prometheus/main.go:396 +0x53cc fp=0xc00092bf88 sp=0xc00092b3d8 > pc=0x21f9a0c > runtime.main() > /usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc00092bfe0 > sp=0xc00092bf88 pc=0x43a0a9 > runtime.goexit() > /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00092bfe8 > sp=0xc00092bfe0 pc=0x46ffa1 > > goroutine 9 [select]: > go.opencensus.io/stats/view.(*worker).start(0xc00030c280) > /app/vendor/go.opencensus.io/stats/view/worker.go:276 +0x105 > created by go.opencensus.io/stats/view.init.0 > /app/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68 > > goroutine 85 [select]: > github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc000276780) > /app/pkg/logging/dedupe.go:75 +0x1e5 > created by github.com/prometheus/prometheus/pkg/logging.Dedupe > /app/pkg/logging/dedupe.go:61 +0xcf > > goroutine 92 [chan receive]: > > github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc00018ad20) > /app/storage/remote/write.go:96 +0xb6 > created by github.com/prometheus/prometheus/storage/remote.NewWriteStorage > /app/storage/remote/write.go:89 +0x305 > > ============================================================================ > level=info ts=2021-01-19T05:42:16.710Z caller=main.go:360 msg="Starting > Prometheus" version="(version=2.23.0, branch=HEAD, > revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > level=info ts=2021-01-19T05:42:16.711Z caller=main.go:365 > build_context="(go=go1.15.5, user=root@37609b3a0a21, > date=20201126-10:56:17)" > level=info ts=2021-01-19T05:42:16.711Z caller=main.go:366 > host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 > 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > level=info ts=2021-01-19T05:42:16.711Z caller=main.go:367 > fd_limits="(soft=1048576, hard=1048576)" > level=info ts=2021-01-19T05:42:16.711Z caller=main.go:368 > vm_limits="(soft=unlimited, hard=unlimited)" > level=info ts=2021-01-19T05:42:16.796Z caller=web.go:528 component=web > msg="Start listening for connections" address=0.0.0.0:9090 > level=info ts=2021-01-19T05:42:16.796Z caller=main.go:722 msg="Starting > TSDB ..." > level=info ts=2021-01-19T05:42:16.844Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610695556030 maxt=1610697600000 > ulid=01EW2S9SMSDCR35G93DT27G6NR > level=info ts=2021-01-19T05:42:16.875Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610697600000 maxt=1610704800000 > ulid=01EW2V7K5TDBRGNWYZXMMN50C7 > level=info ts=2021-01-19T05:42:16.901Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610704800000 maxt=1610712000000 > ulid=01EW323KD7FYEMJ0V03RX7TDR7 > level=info ts=2021-01-19T05:42:16.925Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610733600000 maxt=1610740800000 > ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR > level=info ts=2021-01-19T05:42:16.951Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610712000000 maxt=1610733600000 > ulid=01EW3XJEZR86TBBDJFVTBNZ8VF > level=info ts=2021-01-19T05:42:16.976Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610740800000 maxt=1610748000000 > ulid=01EW44G303K0DNBB2EDM8DX69E > level=info ts=2021-01-19T05:42:17.001Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610748000000 maxt=1610755200000 > ulid=01EW4B9NW6T794HCZATQJZJ4SZ > level=info ts=2021-01-19T05:42:17.049Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610755200000 maxt=1610762400000 > ulid=01EW4J689JDY2GWPCPDNHD0DNJ > level=info ts=2021-01-19T05:42:17.074Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610762400000 maxt=1610769600000 > ulid=01EW4S14CHH73BZHP8152M3594 > level=info ts=2021-01-19T05:42:17.099Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610769600000 maxt=1610776800000 > ulid=01EW4ZXAASHGVGEY6QZ4C6R47M > level=info ts=2021-01-19T05:42:17.125Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610798400000 maxt=1610805600000 > ulid=01EW5VBMBQBYV98Q278AHHNK70 > level=info ts=2021-01-19T05:42:17.155Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610776800000 maxt=1610798400000 > ulid=01EW5VBP3B7KSWNETV0AEKFSWY > level=info ts=2021-01-19T05:42:17.184Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610805600000 maxt=1610812800000 > ulid=01EW627M0DHYWTCC0N237SC62D > level=info ts=2021-01-19T05:42:17.234Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610812800000 maxt=1610820000000 > ulid=01EW6939M4CNXWMHGT34SD5N6R > level=info ts=2021-01-19T05:42:17.259Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610820000000 maxt=1610841600000 > ulid=01EW74JPT1T99G30CWTAE8YTVC > level=info ts=2021-01-19T05:42:17.285Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610863200000 maxt=1610870400000 > ulid=01EW7S59W18PRNC14KHMY2NQDF > level=info ts=2021-01-19T05:42:17.311Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610841600000 maxt=1610863200000 > ulid=01EW7S5CEZAKV5FHJD1BYN7VFS > level=info ts=2021-01-19T05:42:17.337Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610870400000 maxt=1610877600000 > ulid=01EW8012X70MJZYVQTEKERJSF0 > level=info ts=2021-01-19T05:42:17.393Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610877600000 maxt=1610884800000 > ulid=01EW86X36BN7KDN3YQY4KH1X7Z > level=info ts=2021-01-19T05:42:17.418Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610884800000 maxt=1610892000000 > ulid=01EW8DRFQEXQP7TCA6W9ZCWJJC > level=info ts=2021-01-19T05:42:17.443Z caller=repair.go:57 component=tsdb > msg="Found healthy block" mint=1610863200000 maxt=1610884800000 > ulid=01EW8DRHV7JX5CV9YKDFYBQQ99 > level=info ts=2021-01-19T05:42:17.606Z caller=main.go:595 msg="Stopping > scrape discovery manager..." > level=info ts=2021-01-19T05:42:17.607Z caller=main.go:609 msg="Stopping > notify discovery manager..." > level=info ts=2021-01-19T05:42:17.607Z caller=main.go:631 msg="Stopping > scrape manager..." > level=info ts=2021-01-19T05:42:17.607Z caller=main.go:605 msg="Notify > discovery manager stopped" > level=info ts=2021-01-19T05:42:17.607Z caller=main.go:591 msg="Scrape > discovery manager stopped" > level=info ts=2021-01-19T05:42:17.607Z caller=manager.go:924 > component="rule manager" msg="Stopping rule manager..." > level=info ts=2021-01-19T05:42:17.608Z caller=manager.go:934 > component="rule manager" msg="Rule manager stopped" > level=info ts=2021-01-19T05:42:17.608Z caller=notifier.go:601 > component=notifier msg="Stopping notification manager..." > level=info ts=2021-01-19T05:42:17.608Z caller=main.go:799 msg="Notifier > manager stopped" > level=info ts=2021-01-19T05:42:17.608Z caller=main.go:625 msg="Scrape > manager stopped" > level=error ts=2021-01-19T05:42:17.608Z caller=main.go:808 err="opening > storage failed: found unsequential head chunk files > /prometheus/chunks_head/000007 (index: 7) and > /prometheus/chunks_head/000027 (index: 27)" > ======================================= > level=info ts=2021-01-19T10:55:57.643Z caller=main.go:337 msg="Starting > Prometheus" version="(version=2.19.1, branch=HEAD, > revision=eba3fdcbf0d378b66600281903e3aab515732b39)" > level=info ts=2021-01-19T10:55:57.644Z caller=main.go:338 > build_context="(go=go1.14.4, user=root@62700b3d0ef9, > date=20200618-16:35:26)" > level=info ts=2021-01-19T10:55:57.644Z caller=main.go:339 > host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 > 09:13:52 UTC 2020 x86_64 prometheus-84fb48b4d9-ddq9m (none))" > level=info ts=2021-01-19T10:55:57.644Z caller=main.go:340 > fd_limits="(soft=1048576, hard=1048576)" > level=info ts=2021-01-19T10:55:57.644Z caller=main.go:341 > vm_limits="(soft=unlimited, hard=unlimited)" > level=error ts=2021-01-19T10:55:57.704Z caller=query_logger.go:93 > component=activeQueryTracker msg="Error setting filesize." filesize=20001 > err="truncate /prometheus/queries.active: interrupted system call" > panic: Unable to create mmap-ed active query log > > goroutine 1 [running]: > > github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff2f2aa74d, > 0xb, 0x14, 0x2f3b7e0, 0xc00074d800, 0x2f3b7e0) > /app/promql/query_logger.go:117 +0x4cd > main.main() > /app/cmd/prometheus/main.go:368 +0x4ed2 > > > Appreciate any help > > Thanks. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/312916b4-8d54-4529-820a-f9b81eb01b1dn%40googlegroups.com.
