I have just setup exactly the same and find the mounting is a bit hit or miss - did you get anywhere with this?
On Tuesday, 19 January 2021 at 13:23:56 UTC Sdh26 wrote: > We are not attaching Prometheus directly with NFS or SMB. We are using > PVC, there Kubernetes connect using protocol. > Container only see as regular volume attached as it is, it is transparent > with Kubernetes as per my understanding. > On Tuesday, January 19, 2021 at 5:39:43 PM UTC+5:30 Sdh26 wrote: > >> Hello , >> >> We are facing an issue with Prometheus which is running on Azure >> Kubernetes Service (v1.19.1). Nodes are Ubuntu 18.04. >> We are using Azure Storage File Share to store the prometheus data. >> The Prometheus pods runs for couple of hours if we clear the data but >> then keeps going to CrashLoopBackOff and throws multiple errors. >> These are the Logs which we have gathered on multiple instances. We were >> using Prometheus v2.19.1 but we tried with Prometheus version 2.23.0 also >> but doesn't improve. >> The issue started since we upgraded the Kubernetes version from 1.17.9 to >> 1.19.1. >> Is there any compatibility issue during upgrade? >> =============================================== >> >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:360 msg="Starting >> Prometheus" version="(version=2.23.0, branch=HEAD, >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:365 >> build_context="(go=go1.15.5, user=root@37609b3a0a21, >> date=20201126-10:56:17)" >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:366 >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:367 >> fd_limits="(soft=1048576, hard=1048576)" >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:368 >> vm_limits="(soft=unlimited, hard=unlimited)" >> level=info ts=2021-01-15T07:25:31.526Z caller=main.go:722 msg="Starting >> TSDB ..." >> level=info ts=2021-01-15T07:25:31.526Z caller=web.go:528 component=web >> msg="Start listening for connections" address=0.0.0.0:9090 >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:645 component=tsdb >> msg="Replaying on-disk memory mappable chunks if any" >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:659 component=tsdb >> msg="On-disk memory mappable chunks replay completed" duration=4.5µs >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:665 component=tsdb >> msg="Replaying WAL, this may take a while" >> level=info ts=2021-01-15T07:25:31.824Z caller=head.go:717 component=tsdb >> msg="WAL segment loaded" segment=0 maxSegment=0 >> level=info ts=2021-01-15T07:25:31.824Z caller=head.go:722 component=tsdb >> msg="WAL replay completed" checkpoint_replay_duration=23.473299ms >> wal_replay_duration=43.082181ms total_replay_duration=66.57608ms >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 fs_type=fe534d42 >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:745 msg="TSDB >> started" >> level=info ts=2021-01-15T07:25:31.862Z caller=main.go:871 msg="Loading >> configuration file" filename=/etc/prometheus/prometheus.yml >> level=info ts=2021-01-15T07:25:31.950Z caller=main.go:902 msg="Completed >> loading of configuration file" filename=/etc/prometheus/prometheus.yml >> totalDuration=88.228971ms remote_storage=2.901µs web_handler=500ns >> query_engine=1.2µs scrape=380.501µs scrape_sd=135.301µs notify=27.1µs >> notify_sd=14.6µs rules=67.262483ms >> level=info ts=2021-01-15T07:25:31.950Z caller=main.go:694 msg="Server is >> ready to receive web requests." >> level=error ts=2021-01-15T07:52:33.026Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T08:18:36.113Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T08:21:38.206Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T08:24:40.278Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T08:40:42.841Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T09:50:47.814Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T10:26:03.298Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: populate >> block: write chunks: preallocate: seek >> /prometheus/01EW2S98EFR81VFT0677NH81ZF.tmp-for-creation/chunks/000001.tmp: >> interrupted system call" >> level=error ts=2021-01-15T10:26:05.816Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: populate >> block: write chunks: preallocate: seek >> /prometheus/01EW2S9AXV1E42NHPGV0K9Z2MC.tmp-for-creation/chunks/000001.tmp: >> interrupted system call" >> level=error ts=2021-01-15T10:26:15.075Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: populate >> block: write chunks: preallocate: seek >> /prometheus/01EW2S9KZ7GZZ1N36NFMNDCKRV.tmp-for-creation/chunks/000001.tmp: >> interrupted system call" >> level=info ts=2021-01-15T10:26:21.209Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610695556030 maxt=1610697600000 >> ulid=01EW2S9SMSDCR35G93DT27G6NR duration=767.820675ms >> level=info ts=2021-01-15T10:26:21.412Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=25.718613ms >> level=error ts=2021-01-15T10:36:22.279Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T10:55:25.565Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T10:56:27.577Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=info ts=2021-01-15T11:00:06.243Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610697600000 maxt=1610704800000 >> ulid=01EW2V7K5TDBRGNWYZXMMN50C7 duration=808.923904ms >> level=info ts=2021-01-15T11:00:06.487Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=33.110139ms >> level=error ts=2021-01-15T11:48:11.574Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T11:55:14.296Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T12:09:17.495Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T12:24:20.974Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T12:51:25.739Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T12:55:28.056Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T13:00:02.893Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: open chunk >> writer: mkdir >> /prometheus/01EW3237WF9WWH4TDY3HA2KBRK.tmp-for-creation/chunks: interrupted >> system call" >> level=error ts=2021-01-15T13:00:05.471Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: open chunk >> writer: mkdir >> /prometheus/01EW323AD1YAQ6VXB2G6Q6HZ58.tmp-for-creation/chunks: interrupted >> system call" >> level=info ts=2021-01-15T13:00:15.444Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610704800000 maxt=1610712000000 >> ulid=01EW323KD7FYEMJ0V03RX7TDR7 duration=812.389067ms >> level=info ts=2021-01-15T13:00:15.725Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=31.144829ms >> level=error ts=2021-01-15T13:07:16.892Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T13:35:22.717Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T13:42:25.587Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T13:47:28.162Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T13:55:31.084Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T14:19:36.277Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T14:24:38.878Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T14:54:45.004Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T14:56:47.144Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T15:00:06.054Z caller=compact.go:541 >> component=tsdb msg="removed tmp folder after failed compaction" >> err="unlinkat /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation: >> directory not empty" >> level=error ts=2021-01-15T15:00:06.054Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: sync >> /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation/index: interrupted >> system call" >> level=info ts=2021-01-15T15:00:07.791Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610712000000 maxt=1610719200000 >> ulid=01EW38Z38E2TA8CDEJNJ130QV9 duration=736.679699ms >> level=info ts=2021-01-15T15:00:08.081Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=27.759017ms >> level=info ts=2021-01-15T15:00:08.261Z caller=checkpoint.go:95 >> component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 >> mint=1610719200000 >> level=info ts=2021-01-15T15:00:08.689Z caller=head.go:908 component=tsdb >> msg="WAL checkpoint complete" first=0 last=1 duration=454.463412ms >> level=error ts=2021-01-15T15:01:08.914Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T15:17:13.515Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T15:44:19.858Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T15:50:22.808Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T15:54:25.360Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T15:55:27.371Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T16:02:32.493Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T16:09:35.501Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T16:17:38.646Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T16:19:40.796Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T16:28:44.071Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=info ts=2021-01-15T17:00:01.827Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610719200000 maxt=1610726400000 >> ulid=01EW3FTMKQYF12V439SBT6B17P duration=811.71891ms >> level=info ts=2021-01-15T17:00:02.175Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=28.805321ms >> level=error ts=2021-01-15T17:00:03.791Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact >> [/prometheus/01EW2S9SMSDCR35G93DT27G6NR >> /prometheus/01EW2V7K5TDBRGNWYZXMMN50C7 >> /prometheus/01EW323KD7FYEMJ0V03RX7TDR7]: write merged meta: rename >> /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json.tmp >> /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json: >> interrupted system call" >> level=error ts=2021-01-15T17:01:05.830Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T17:03:10.067Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T17:18:14.829Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T17:24:17.637Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T17:36:21.558Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T17:38:23.743Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T17:48:27.143Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T18:12:32.875Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T18:20:36.002Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T18:24:38.461Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T18:32:41.690Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T18:36:44.188Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T18:42:47.011Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T19:00:06.007Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: write merged >> meta: rename >> /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json.tmp >> /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json: >> interrupted system call" >> level=info ts=2021-01-15T19:00:07.649Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610726400000 maxt=1610733600000 >> ulid=01EW3PPHPZ6ZZM2JGKF69F96RN duration=641.827141ms >> level=info ts=2021-01-15T19:00:07.932Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=24.2159ms >> level=info ts=2021-01-15T19:00:08.096Z caller=checkpoint.go:95 >> component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 >> mint=1610733600000 >> level=info ts=2021-01-15T19:00:08.589Z caller=head.go:908 component=tsdb >> msg="WAL checkpoint complete" first=2 last=3 duration=516.496925ms >> level=error ts=2021-01-15T19:08:09.957Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T19:16:13.284Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T19:19:15.693Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T19:20:17.699Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T19:39:24.830Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T19:42:27.215Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T19:45:29.593Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T19:56:33.420Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:06:37.228Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:08:39.420Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:19:43.242Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:30:47.040Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:35:49.798Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:38:52.166Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:49:55.966Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:50:57.973Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T20:59:03.232Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T21:00:05.262Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T21:00:07.878Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: write merged >> meta: rename >> /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json.tmp >> /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json: >> interrupted system call" >> level=info ts=2021-01-15T21:00:12.523Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610733600000 maxt=1610740800000 >> ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR duration=644.683302ms >> level=info ts=2021-01-15T21:00:12.828Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=22.771295ms >> level=info ts=2021-01-15T21:00:13.960Z caller=compact.go:443 >> component=tsdb msg="compact blocks" count=3 mint=1610712000000 >> maxt=1610733600000 ulid=01EW3XJEZR86TBBDJFVTBNZ8VF >> sources="[01EW38Z38E2TA8CDEJNJ130QV9 01EW3FTMKQYF12V439SBT6B17P >> 01EW3PPHPZ6ZZM2JGKF69F96RN]" duration=827.624569ms >> level=info ts=2021-01-15T21:00:14.381Z caller=db.go:1187 component=tsdb >> msg="Deleting obsolete block" block=01EW38Z38E2TA8CDEJNJ130QV9 >> level=info ts=2021-01-15T21:00:14.547Z caller=db.go:1187 component=tsdb >> msg="Deleting obsolete block" block=01EW3FTMKQYF12V439SBT6B17P >> level=info ts=2021-01-15T21:00:14.705Z caller=db.go:1187 component=tsdb >> msg="Deleting obsolete block" block=01EW3PPHPZ6ZZM2JGKF69F96RN >> level=error ts=2021-01-15T21:03:15.152Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T21:20:19.710Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T21:36:24.045Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T21:38:26.227Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T21:46:29.339Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T21:47:31.378Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T22:00:37.302Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T22:02:39.487Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T22:17:43.600Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T22:26:47.082Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T22:35:50.353Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T22:36:52.383Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T23:00:06.094Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: write new >> tombstones file: rename >> /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones.tmp >> /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones: >> interrupted system call" >> level=error ts=2021-01-15T23:00:07.633Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: write merged >> meta: rename >> /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json.tmp >> /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json: >> interrupted system call" >> level=error ts=2021-01-15T23:00:13.695Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: rename block >> dir: rename /prometheus/01EW44E60NC3EGCETC25GBW492.tmp-for-creation >> /prometheus/01EW44E60NC3EGCETC25GBW492: interrupted system call" >> level=error ts=2021-01-15T23:00:18.059Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: populate >> block: write chunks: preallocate: seek >> /prometheus/01EW44EAMZKJ5JJ2E42WCM8V0Z.tmp-for-creation/chunks/000001.tmp: >> interrupted system call" >> level=error ts=2021-01-15T23:00:26.689Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: write merged >> meta: rename >> /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json.tmp >> /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json: >> interrupted system call" >> level=error ts=2021-01-15T23:00:43.395Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: write new >> tombstones file: rename >> /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones.tmp >> /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones: >> interrupted system call" >> level=info ts=2021-01-15T23:01:16.055Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610740800000 maxt=1610748000000 >> ulid=01EW44G303K0DNBB2EDM8DX69E duration=659.365701ms >> level=info ts=2021-01-15T23:01:16.353Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=23.148498ms >> level=info ts=2021-01-15T23:01:16.514Z caller=checkpoint.go:95 >> component=tsdb msg="Creating checkpoint" from_segment=4 to_segment=5 >> mint=1610748000000 >> level=info ts=2021-01-15T23:01:17.004Z caller=head.go:908 component=tsdb >> msg="WAL checkpoint complete" first=4 last=5 duration=512.755478ms >> level=error ts=2021-01-15T23:22:20.652Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T23:25:23.109Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T23:40:27.567Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-15T23:50:31.230Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T00:03:35.502Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T00:20:40.429Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T00:21:42.458Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T00:23:46.659Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T00:25:48.862Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T00:37:52.910Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T00:57:58.400Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T01:00:00.602Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=info ts=2021-01-16T01:00:06.068Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610748000000 maxt=1610755200000 >> ulid=01EW4B9NW6T794HCZATQJZJ4SZ duration=686.024281ms >> level=info ts=2021-01-16T01:00:06.429Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=22.475094ms >> level=error ts=2021-01-16T01:00:06.563Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: head memory truncate: truncate >> chunks.HeadReadWriter: remove /prometheus/chunks_head/000007: interrupted >> system call" >> level=error ts=2021-01-16T01:08:12.075Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T01:15:15.300Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T01:20:18.133Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T01:22:20.357Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T01:31:24.051Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T01:50:29.872Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T01:52:32.155Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T02:00:35.607Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T02:40:45.873Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T02:54:50.472Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="plan compaction: readdirent: no such file or >> directory" >> level=error ts=2021-01-16T03:00:06.014Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: write merged >> meta: rename >> /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json.tmp >> /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json: >> interrupted system call" >> level=error ts=2021-01-16T03:00:16.046Z caller=compact.go:541 >> component=tsdb msg="removed tmp folder after failed compaction" >> err="unlinkat /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation: >> directory not empty" >> level=error ts=2021-01-16T03:00:16.046Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: sync >> /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation/index: interrupted >> system call" >> level=error ts=2021-01-16T03:00:20.488Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: open chunk >> writer: mkdir >> /prometheus/01EW4J5VTSRT3KXABDFXETHK99.tmp-for-creation/chunks: interrupted >> system call" >> level=error ts=2021-01-16T03:00:25.202Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact head: persist head block: write new >> tombstones file: rename >> /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones.tmp >> /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones: >> interrupted system call" >> level=info ts=2021-01-16T03:00:33.853Z caller=compact.go:502 >> component=tsdb msg="write block" mint=1610755200000 maxt=1610762400000 >> ulid=01EW4J689JDY2GWPCPDNHD0DNJ duration=650.542029ms >> level=info ts=2021-01-16T03:00:34.211Z caller=head.go:824 component=tsdb >> msg="Head GC completed" duration=22.278794ms >> level=error ts=2021-01-16T03:00:35.654Z caller=db.go:746 component=tsdb >> msg="compaction failed" err="compact >> [/prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR >> /prometheus/01EW44G303K0DNBB2EDM8DX69E >> /prometheus/01EW4B9NW6T794HCZATQJZJ4SZ]: 2 errors: rename block dir: rename >> /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK.tmp-for-creation >> /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK: interrupted system call; setting >> compaction failed for block: /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR: rename >> /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json.tmp >> /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json: interrupted system call" >> >> >> =============================== >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:360 msg="Starting >> Prometheus" version="(version=2.23.0, branch=HEAD, >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:365 >> build_context="(go=go1.15.5, user=root@37609b3a0a21, >> date=20201126-10:56:17)" >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:366 >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:367 >> fd_limits="(soft=1048576, hard=1048576)" >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:368 >> vm_limits="(soft=unlimited, hard=unlimited)" >> level=error ts=2021-01-19T05:06:08.726Z caller=query_logger.go:71 >> component=activeQueryTracker msg="Failed to read query log file" err=EOF >> unexpected fault address 0x7f3c06738000 >> fatal error: fault >> [signal SIGBUS: bus error code=0x2 addr=0x7f3c06738000 pc=0x470ea8] >> >> goroutine 1 [running]: >> runtime.throw(0x29f4a25, 0x5) >> /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00092b248 >> sp=0xc00092b218 pc=0x4378b2 >> runtime.sigpanic() >> /usr/local/go/src/runtime/signal_unix.go:739 +0x485 fp=0xc00092b278 >> sp=0xc00092b248 pc=0x44e085 >> runtime.memmove(0x7f3c06738000, 0x29f1e48, 0x1) >> /usr/local/go/src/runtime/memmove_amd64.s:148 +0x108 fp=0xc00092b280 >> sp=0xc00092b278 pc=0x470ea8 >> >> github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff842a774d, >> >> 0xb, 0x14, 0x30d9a40, 0xc00077ede0, 0x30d9a40) >> /app/promql/query_logger.go:120 +0x3a5 fp=0xc00092b3d8 sp=0xc00092b280 >> pc=0x20fac05 >> main.main() >> /app/cmd/prometheus/main.go:396 +0x53cc fp=0xc00092bf88 sp=0xc00092b3d8 >> pc=0x21f9a0c >> runtime.main() >> /usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc00092bfe0 >> sp=0xc00092bf88 pc=0x43a0a9 >> runtime.goexit() >> /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00092bfe8 >> sp=0xc00092bfe0 pc=0x46ffa1 >> >> goroutine 9 [select]: >> go.opencensus.io/stats/view.(*worker).start(0xc00030c280) >> /app/vendor/go.opencensus.io/stats/view/worker.go:276 +0x105 >> created by go.opencensus.io/stats/view.init.0 >> /app/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68 >> >> goroutine 85 [select]: >> github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc000276780) >> /app/pkg/logging/dedupe.go:75 +0x1e5 >> created by github.com/prometheus/prometheus/pkg/logging.Dedupe >> /app/pkg/logging/dedupe.go:61 +0xcf >> >> goroutine 92 [chan receive]: >> >> github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc00018ad20) >> /app/storage/remote/write.go:96 +0xb6 >> created by >> github.com/prometheus/prometheus/storage/remote.NewWriteStorage >> /app/storage/remote/write.go:89 +0x305 >> >> ============================================================================ >> level=info ts=2021-01-19T05:42:16.710Z caller=main.go:360 msg="Starting >> Prometheus" version="(version=2.23.0, branch=HEAD, >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:365 >> build_context="(go=go1.15.5, user=root@37609b3a0a21, >> date=20201126-10:56:17)" >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:366 >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:367 >> fd_limits="(soft=1048576, hard=1048576)" >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:368 >> vm_limits="(soft=unlimited, hard=unlimited)" >> level=info ts=2021-01-19T05:42:16.796Z caller=web.go:528 component=web >> msg="Start listening for connections" address=0.0.0.0:9090 >> level=info ts=2021-01-19T05:42:16.796Z caller=main.go:722 msg="Starting >> TSDB ..." >> level=info ts=2021-01-19T05:42:16.844Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610695556030 maxt=1610697600000 >> ulid=01EW2S9SMSDCR35G93DT27G6NR >> level=info ts=2021-01-19T05:42:16.875Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610697600000 maxt=1610704800000 >> ulid=01EW2V7K5TDBRGNWYZXMMN50C7 >> level=info ts=2021-01-19T05:42:16.901Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610704800000 maxt=1610712000000 >> ulid=01EW323KD7FYEMJ0V03RX7TDR7 >> level=info ts=2021-01-19T05:42:16.925Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610733600000 maxt=1610740800000 >> ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR >> level=info ts=2021-01-19T05:42:16.951Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610712000000 maxt=1610733600000 >> ulid=01EW3XJEZR86TBBDJFVTBNZ8VF >> level=info ts=2021-01-19T05:42:16.976Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610740800000 maxt=1610748000000 >> ulid=01EW44G303K0DNBB2EDM8DX69E >> level=info ts=2021-01-19T05:42:17.001Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610748000000 maxt=1610755200000 >> ulid=01EW4B9NW6T794HCZATQJZJ4SZ >> level=info ts=2021-01-19T05:42:17.049Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610755200000 maxt=1610762400000 >> ulid=01EW4J689JDY2GWPCPDNHD0DNJ >> level=info ts=2021-01-19T05:42:17.074Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610762400000 maxt=1610769600000 >> ulid=01EW4S14CHH73BZHP8152M3594 >> level=info ts=2021-01-19T05:42:17.099Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610769600000 maxt=1610776800000 >> ulid=01EW4ZXAASHGVGEY6QZ4C6R47M >> level=info ts=2021-01-19T05:42:17.125Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610798400000 maxt=1610805600000 >> ulid=01EW5VBMBQBYV98Q278AHHNK70 >> level=info ts=2021-01-19T05:42:17.155Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610776800000 maxt=1610798400000 >> ulid=01EW5VBP3B7KSWNETV0AEKFSWY >> level=info ts=2021-01-19T05:42:17.184Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610805600000 maxt=1610812800000 >> ulid=01EW627M0DHYWTCC0N237SC62D >> level=info ts=2021-01-19T05:42:17.234Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610812800000 maxt=1610820000000 >> ulid=01EW6939M4CNXWMHGT34SD5N6R >> level=info ts=2021-01-19T05:42:17.259Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610820000000 maxt=1610841600000 >> ulid=01EW74JPT1T99G30CWTAE8YTVC >> level=info ts=2021-01-19T05:42:17.285Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610863200000 maxt=1610870400000 >> ulid=01EW7S59W18PRNC14KHMY2NQDF >> level=info ts=2021-01-19T05:42:17.311Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610841600000 maxt=1610863200000 >> ulid=01EW7S5CEZAKV5FHJD1BYN7VFS >> level=info ts=2021-01-19T05:42:17.337Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610870400000 maxt=1610877600000 >> ulid=01EW8012X70MJZYVQTEKERJSF0 >> level=info ts=2021-01-19T05:42:17.393Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610877600000 maxt=1610884800000 >> ulid=01EW86X36BN7KDN3YQY4KH1X7Z >> level=info ts=2021-01-19T05:42:17.418Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610884800000 maxt=1610892000000 >> ulid=01EW8DRFQEXQP7TCA6W9ZCWJJC >> level=info ts=2021-01-19T05:42:17.443Z caller=repair.go:57 component=tsdb >> msg="Found healthy block" mint=1610863200000 maxt=1610884800000 >> ulid=01EW8DRHV7JX5CV9YKDFYBQQ99 >> level=info ts=2021-01-19T05:42:17.606Z caller=main.go:595 msg="Stopping >> scrape discovery manager..." >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:609 msg="Stopping >> notify discovery manager..." >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:631 msg="Stopping >> scrape manager..." >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:605 msg="Notify >> discovery manager stopped" >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:591 msg="Scrape >> discovery manager stopped" >> level=info ts=2021-01-19T05:42:17.607Z caller=manager.go:924 >> component="rule manager" msg="Stopping rule manager..." >> level=info ts=2021-01-19T05:42:17.608Z caller=manager.go:934 >> component="rule manager" msg="Rule manager stopped" >> level=info ts=2021-01-19T05:42:17.608Z caller=notifier.go:601 >> component=notifier msg="Stopping notification manager..." >> level=info ts=2021-01-19T05:42:17.608Z caller=main.go:799 msg="Notifier >> manager stopped" >> level=info ts=2021-01-19T05:42:17.608Z caller=main.go:625 msg="Scrape >> manager stopped" >> level=error ts=2021-01-19T05:42:17.608Z caller=main.go:808 err="opening >> storage failed: found unsequential head chunk files >> /prometheus/chunks_head/000007 (index: 7) and >> /prometheus/chunks_head/000027 (index: 27)" >> ======================================= >> level=info ts=2021-01-19T10:55:57.643Z caller=main.go:337 msg="Starting >> Prometheus" version="(version=2.19.1, branch=HEAD, >> revision=eba3fdcbf0d378b66600281903e3aab515732b39)" >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:338 >> build_context="(go=go1.14.4, user=root@62700b3d0ef9, >> date=20200618-16:35:26)" >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:339 >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 >> 09:13:52 UTC 2020 x86_64 prometheus-84fb48b4d9-ddq9m (none))" >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:340 >> fd_limits="(soft=1048576, hard=1048576)" >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:341 >> vm_limits="(soft=unlimited, hard=unlimited)" >> level=error ts=2021-01-19T10:55:57.704Z caller=query_logger.go:93 >> component=activeQueryTracker msg="Error setting filesize." filesize=20001 >> err="truncate /prometheus/queries.active: interrupted system call" >> panic: Unable to create mmap-ed active query log >> >> goroutine 1 [running]: >> >> github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff2f2aa74d, >> >> 0xb, 0x14, 0x2f3b7e0, 0xc00074d800, 0x2f3b7e0) >> /app/promql/query_logger.go:117 +0x4cd >> main.main() >> /app/cmd/prometheus/main.go:368 +0x4ed2 >> >> >> Appreciate any help >> >> Thanks. >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/dd2528cb-522d-44ac-900a-af2a53ad431bn%40googlegroups.com.
