>> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 >> fs_type=fe534d42
That line indicates that this is using SMB - which is a network file system not supported by Prometheus. On 22 Feb 12:16, Adam Carter wrote: > I have just setup exactly the same and find the mounting is a bit hit or > miss - did you get anywhere with this? > > > On Tuesday, 19 January 2021 at 13:23:56 UTC Sdh26 wrote: > > > We are not attaching Prometheus directly with NFS or SMB. We are using > > PVC, there Kubernetes connect using protocol. > > Container only see as regular volume attached as it is, it is transparent > > with Kubernetes as per my understanding. > > On Tuesday, January 19, 2021 at 5:39:43 PM UTC+5:30 Sdh26 wrote: > > > >> Hello , > >> > >> We are facing an issue with Prometheus which is running on Azure > >> Kubernetes Service (v1.19.1). Nodes are Ubuntu 18.04. > >> We are using Azure Storage File Share to store the prometheus data. > >> The Prometheus pods runs for couple of hours if we clear the data but > >> then keeps going to CrashLoopBackOff and throws multiple errors. > >> These are the Logs which we have gathered on multiple instances. We were > >> using Prometheus v2.19.1 but we tried with Prometheus version 2.23.0 also > >> but doesn't improve. > >> The issue started since we upgraded the Kubernetes version from 1.17.9 to > >> 1.19.1. > >> Is there any compatibility issue during upgrade? > >> =============================================== > >> > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:360 msg="Starting > >> Prometheus" version="(version=2.23.0, branch=HEAD, > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:365 > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, > >> date=20201126-10:56:17)" > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:366 > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:367 > >> fd_limits="(soft=1048576, hard=1048576)" > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:368 > >> vm_limits="(soft=unlimited, hard=unlimited)" > >> level=info ts=2021-01-15T07:25:31.526Z caller=main.go:722 msg="Starting > >> TSDB ..." > >> level=info ts=2021-01-15T07:25:31.526Z caller=web.go:528 component=web > >> msg="Start listening for connections" address=0.0.0.0:9090 > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:645 component=tsdb > >> msg="Replaying on-disk memory mappable chunks if any" > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:659 component=tsdb > >> msg="On-disk memory mappable chunks replay completed" duration=4.5盜 > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:665 component=tsdb > >> msg="Replaying WAL, this may take a while" > >> level=info ts=2021-01-15T07:25:31.824Z caller=head.go:717 component=tsdb > >> msg="WAL segment loaded" segment=0 maxSegment=0 > >> level=info ts=2021-01-15T07:25:31.824Z caller=head.go:722 component=tsdb > >> msg="WAL replay completed" checkpoint_replay_duration=23.473299ms > >> wal_replay_duration=43.082181ms total_replay_duration=66.57608ms > >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 fs_type=fe534d42 > >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:745 msg="TSDB > >> started" > >> level=info ts=2021-01-15T07:25:31.862Z caller=main.go:871 msg="Loading > >> configuration file" filename=/etc/prometheus/prometheus.yml > >> level=info ts=2021-01-15T07:25:31.950Z caller=main.go:902 msg="Completed > >> loading of configuration file" filename=/etc/prometheus/prometheus.yml > >> totalDuration=88.228971ms remote_storage=2.901盜 web_handler=500ns > >> query_engine=1.2盜 scrape=380.501盜 scrape_sd=135.301盜 notify=27.1盜 > >> notify_sd=14.6盜 rules=67.262483ms > >> level=info ts=2021-01-15T07:25:31.950Z caller=main.go:694 msg="Server is > >> ready to receive web requests." > >> level=error ts=2021-01-15T07:52:33.026Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T08:18:36.113Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T08:21:38.206Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T08:24:40.278Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T08:40:42.841Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T09:50:47.814Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T10:26:03.298Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: populate > >> block: write chunks: preallocate: seek > >> /prometheus/01EW2S98EFR81VFT0677NH81ZF.tmp-for-creation/chunks/000001.tmp: > >> interrupted system call" > >> level=error ts=2021-01-15T10:26:05.816Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: populate > >> block: write chunks: preallocate: seek > >> /prometheus/01EW2S9AXV1E42NHPGV0K9Z2MC.tmp-for-creation/chunks/000001.tmp: > >> interrupted system call" > >> level=error ts=2021-01-15T10:26:15.075Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: populate > >> block: write chunks: preallocate: seek > >> /prometheus/01EW2S9KZ7GZZ1N36NFMNDCKRV.tmp-for-creation/chunks/000001.tmp: > >> interrupted system call" > >> level=info ts=2021-01-15T10:26:21.209Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610695556030 maxt=1610697600000 > >> ulid=01EW2S9SMSDCR35G93DT27G6NR duration=767.820675ms > >> level=info ts=2021-01-15T10:26:21.412Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=25.718613ms > >> level=error ts=2021-01-15T10:36:22.279Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T10:55:25.565Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T10:56:27.577Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=info ts=2021-01-15T11:00:06.243Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610697600000 maxt=1610704800000 > >> ulid=01EW2V7K5TDBRGNWYZXMMN50C7 duration=808.923904ms > >> level=info ts=2021-01-15T11:00:06.487Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=33.110139ms > >> level=error ts=2021-01-15T11:48:11.574Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T11:55:14.296Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T12:09:17.495Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T12:24:20.974Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T12:51:25.739Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T12:55:28.056Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T13:00:02.893Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: open chunk > >> writer: mkdir > >> /prometheus/01EW3237WF9WWH4TDY3HA2KBRK.tmp-for-creation/chunks: > >> interrupted > >> system call" > >> level=error ts=2021-01-15T13:00:05.471Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: open chunk > >> writer: mkdir > >> /prometheus/01EW323AD1YAQ6VXB2G6Q6HZ58.tmp-for-creation/chunks: > >> interrupted > >> system call" > >> level=info ts=2021-01-15T13:00:15.444Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610704800000 maxt=1610712000000 > >> ulid=01EW323KD7FYEMJ0V03RX7TDR7 duration=812.389067ms > >> level=info ts=2021-01-15T13:00:15.725Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=31.144829ms > >> level=error ts=2021-01-15T13:07:16.892Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T13:35:22.717Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T13:42:25.587Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T13:47:28.162Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T13:55:31.084Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T14:19:36.277Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T14:24:38.878Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T14:54:45.004Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T14:56:47.144Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T15:00:06.054Z caller=compact.go:541 > >> component=tsdb msg="removed tmp folder after failed compaction" > >> err="unlinkat /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation: > >> directory not empty" > >> level=error ts=2021-01-15T15:00:06.054Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: sync > >> /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation/index: interrupted > >> system call" > >> level=info ts=2021-01-15T15:00:07.791Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610712000000 maxt=1610719200000 > >> ulid=01EW38Z38E2TA8CDEJNJ130QV9 duration=736.679699ms > >> level=info ts=2021-01-15T15:00:08.081Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=27.759017ms > >> level=info ts=2021-01-15T15:00:08.261Z caller=checkpoint.go:95 > >> component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 > >> mint=1610719200000 > >> level=info ts=2021-01-15T15:00:08.689Z caller=head.go:908 component=tsdb > >> msg="WAL checkpoint complete" first=0 last=1 duration=454.463412ms > >> level=error ts=2021-01-15T15:01:08.914Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T15:17:13.515Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T15:44:19.858Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T15:50:22.808Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T15:54:25.360Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T15:55:27.371Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T16:02:32.493Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T16:09:35.501Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T16:17:38.646Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T16:19:40.796Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T16:28:44.071Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=info ts=2021-01-15T17:00:01.827Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610719200000 maxt=1610726400000 > >> ulid=01EW3FTMKQYF12V439SBT6B17P duration=811.71891ms > >> level=info ts=2021-01-15T17:00:02.175Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=28.805321ms > >> level=error ts=2021-01-15T17:00:03.791Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact > >> [/prometheus/01EW2S9SMSDCR35G93DT27G6NR > >> /prometheus/01EW2V7K5TDBRGNWYZXMMN50C7 > >> /prometheus/01EW323KD7FYEMJ0V03RX7TDR7]: write merged meta: rename > >> /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json.tmp > >> /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json: > >> interrupted system call" > >> level=error ts=2021-01-15T17:01:05.830Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T17:03:10.067Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T17:18:14.829Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T17:24:17.637Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T17:36:21.558Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T17:38:23.743Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T17:48:27.143Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T18:12:32.875Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T18:20:36.002Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T18:24:38.461Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T18:32:41.690Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T18:36:44.188Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T18:42:47.011Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T19:00:06.007Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: write > >> merged > >> meta: rename > >> /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json.tmp > >> /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json: > >> interrupted system call" > >> level=info ts=2021-01-15T19:00:07.649Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610726400000 maxt=1610733600000 > >> ulid=01EW3PPHPZ6ZZM2JGKF69F96RN duration=641.827141ms > >> level=info ts=2021-01-15T19:00:07.932Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=24.2159ms > >> level=info ts=2021-01-15T19:00:08.096Z caller=checkpoint.go:95 > >> component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 > >> mint=1610733600000 > >> level=info ts=2021-01-15T19:00:08.589Z caller=head.go:908 component=tsdb > >> msg="WAL checkpoint complete" first=2 last=3 duration=516.496925ms > >> level=error ts=2021-01-15T19:08:09.957Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T19:16:13.284Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T19:19:15.693Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T19:20:17.699Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T19:39:24.830Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T19:42:27.215Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T19:45:29.593Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T19:56:33.420Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:06:37.228Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:08:39.420Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:19:43.242Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:30:47.040Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:35:49.798Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:38:52.166Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:49:55.966Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:50:57.973Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T20:59:03.232Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T21:00:05.262Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T21:00:07.878Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: write > >> merged > >> meta: rename > >> /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json.tmp > >> /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json: > >> interrupted system call" > >> level=info ts=2021-01-15T21:00:12.523Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610733600000 maxt=1610740800000 > >> ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR duration=644.683302ms > >> level=info ts=2021-01-15T21:00:12.828Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=22.771295ms > >> level=info ts=2021-01-15T21:00:13.960Z caller=compact.go:443 > >> component=tsdb msg="compact blocks" count=3 mint=1610712000000 > >> maxt=1610733600000 ulid=01EW3XJEZR86TBBDJFVTBNZ8VF > >> sources="[01EW38Z38E2TA8CDEJNJ130QV9 01EW3FTMKQYF12V439SBT6B17P > >> 01EW3PPHPZ6ZZM2JGKF69F96RN]" duration=827.624569ms > >> level=info ts=2021-01-15T21:00:14.381Z caller=db.go:1187 component=tsdb > >> msg="Deleting obsolete block" block=01EW38Z38E2TA8CDEJNJ130QV9 > >> level=info ts=2021-01-15T21:00:14.547Z caller=db.go:1187 component=tsdb > >> msg="Deleting obsolete block" block=01EW3FTMKQYF12V439SBT6B17P > >> level=info ts=2021-01-15T21:00:14.705Z caller=db.go:1187 component=tsdb > >> msg="Deleting obsolete block" block=01EW3PPHPZ6ZZM2JGKF69F96RN > >> level=error ts=2021-01-15T21:03:15.152Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T21:20:19.710Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T21:36:24.045Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T21:38:26.227Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T21:46:29.339Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T21:47:31.378Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T22:00:37.302Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T22:02:39.487Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T22:17:43.600Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T22:26:47.082Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T22:35:50.353Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T22:36:52.383Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T23:00:06.094Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: write new > >> tombstones file: rename > >> /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones.tmp > >> /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones: > >> interrupted system call" > >> level=error ts=2021-01-15T23:00:07.633Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: write > >> merged > >> meta: rename > >> /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json.tmp > >> /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json: > >> interrupted system call" > >> level=error ts=2021-01-15T23:00:13.695Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: rename > >> block > >> dir: rename /prometheus/01EW44E60NC3EGCETC25GBW492.tmp-for-creation > >> /prometheus/01EW44E60NC3EGCETC25GBW492: interrupted system call" > >> level=error ts=2021-01-15T23:00:18.059Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: populate > >> block: write chunks: preallocate: seek > >> /prometheus/01EW44EAMZKJ5JJ2E42WCM8V0Z.tmp-for-creation/chunks/000001.tmp: > >> interrupted system call" > >> level=error ts=2021-01-15T23:00:26.689Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: write > >> merged > >> meta: rename > >> /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json.tmp > >> /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json: > >> interrupted system call" > >> level=error ts=2021-01-15T23:00:43.395Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: write new > >> tombstones file: rename > >> /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones.tmp > >> /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones: > >> interrupted system call" > >> level=info ts=2021-01-15T23:01:16.055Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610740800000 maxt=1610748000000 > >> ulid=01EW44G303K0DNBB2EDM8DX69E duration=659.365701ms > >> level=info ts=2021-01-15T23:01:16.353Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=23.148498ms > >> level=info ts=2021-01-15T23:01:16.514Z caller=checkpoint.go:95 > >> component=tsdb msg="Creating checkpoint" from_segment=4 to_segment=5 > >> mint=1610748000000 > >> level=info ts=2021-01-15T23:01:17.004Z caller=head.go:908 component=tsdb > >> msg="WAL checkpoint complete" first=4 last=5 duration=512.755478ms > >> level=error ts=2021-01-15T23:22:20.652Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T23:25:23.109Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T23:40:27.567Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-15T23:50:31.230Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T00:03:35.502Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T00:20:40.429Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T00:21:42.458Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T00:23:46.659Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T00:25:48.862Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T00:37:52.910Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T00:57:58.400Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T01:00:00.602Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=info ts=2021-01-16T01:00:06.068Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610748000000 maxt=1610755200000 > >> ulid=01EW4B9NW6T794HCZATQJZJ4SZ duration=686.024281ms > >> level=info ts=2021-01-16T01:00:06.429Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=22.475094ms > >> level=error ts=2021-01-16T01:00:06.563Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: head memory truncate: truncate > >> chunks.HeadReadWriter: remove /prometheus/chunks_head/000007: interrupted > >> system call" > >> level=error ts=2021-01-16T01:08:12.075Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T01:15:15.300Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T01:20:18.133Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T01:22:20.357Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T01:31:24.051Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T01:50:29.872Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T01:52:32.155Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T02:00:35.607Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T02:40:45.873Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T02:54:50.472Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="plan compaction: readdirent: no such file or > >> directory" > >> level=error ts=2021-01-16T03:00:06.014Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: write > >> merged > >> meta: rename > >> /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json.tmp > >> /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json: > >> interrupted system call" > >> level=error ts=2021-01-16T03:00:16.046Z caller=compact.go:541 > >> component=tsdb msg="removed tmp folder after failed compaction" > >> err="unlinkat /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation: > >> directory not empty" > >> level=error ts=2021-01-16T03:00:16.046Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: sync > >> /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation/index: interrupted > >> system call" > >> level=error ts=2021-01-16T03:00:20.488Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: open chunk > >> writer: mkdir > >> /prometheus/01EW4J5VTSRT3KXABDFXETHK99.tmp-for-creation/chunks: > >> interrupted > >> system call" > >> level=error ts=2021-01-16T03:00:25.202Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact head: persist head block: write new > >> tombstones file: rename > >> /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones.tmp > >> /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones: > >> interrupted system call" > >> level=info ts=2021-01-16T03:00:33.853Z caller=compact.go:502 > >> component=tsdb msg="write block" mint=1610755200000 maxt=1610762400000 > >> ulid=01EW4J689JDY2GWPCPDNHD0DNJ duration=650.542029ms > >> level=info ts=2021-01-16T03:00:34.211Z caller=head.go:824 component=tsdb > >> msg="Head GC completed" duration=22.278794ms > >> level=error ts=2021-01-16T03:00:35.654Z caller=db.go:746 component=tsdb > >> msg="compaction failed" err="compact > >> [/prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR > >> /prometheus/01EW44G303K0DNBB2EDM8DX69E > >> /prometheus/01EW4B9NW6T794HCZATQJZJ4SZ]: 2 errors: rename block dir: > >> rename > >> /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK.tmp-for-creation > >> /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK: interrupted system call; setting > >> compaction failed for block: /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR: > >> rename > >> /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json.tmp > >> /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json: interrupted system call" > >> > >> > >> =============================== > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:360 msg="Starting > >> Prometheus" version="(version=2.23.0, branch=HEAD, > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:365 > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, > >> date=20201126-10:56:17)" > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:366 > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:367 > >> fd_limits="(soft=1048576, hard=1048576)" > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:368 > >> vm_limits="(soft=unlimited, hard=unlimited)" > >> level=error ts=2021-01-19T05:06:08.726Z caller=query_logger.go:71 > >> component=activeQueryTracker msg="Failed to read query log file" err=EOF > >> unexpected fault address 0x7f3c06738000 > >> fatal error: fault > >> [signal SIGBUS: bus error code=0x2 addr=0x7f3c06738000 pc=0x470ea8] > >> > >> goroutine 1 [running]: > >> runtime.throw(0x29f4a25, 0x5) > >> /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00092b248 > >> sp=0xc00092b218 pc=0x4378b2 > >> runtime.sigpanic() > >> /usr/local/go/src/runtime/signal_unix.go:739 +0x485 fp=0xc00092b278 > >> sp=0xc00092b248 pc=0x44e085 > >> runtime.memmove(0x7f3c06738000, 0x29f1e48, 0x1) > >> /usr/local/go/src/runtime/memmove_amd64.s:148 +0x108 fp=0xc00092b280 > >> sp=0xc00092b278 pc=0x470ea8 > >> > >> github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff842a774d, > >> > >> 0xb, 0x14, 0x30d9a40, 0xc00077ede0, 0x30d9a40) > >> /app/promql/query_logger.go:120 +0x3a5 fp=0xc00092b3d8 sp=0xc00092b280 > >> pc=0x20fac05 > >> main.main() > >> /app/cmd/prometheus/main.go:396 +0x53cc fp=0xc00092bf88 sp=0xc00092b3d8 > >> pc=0x21f9a0c > >> runtime.main() > >> /usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc00092bfe0 > >> sp=0xc00092bf88 pc=0x43a0a9 > >> runtime.goexit() > >> /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00092bfe8 > >> sp=0xc00092bfe0 pc=0x46ffa1 > >> > >> goroutine 9 [select]: > >> go.opencensus.io/stats/view.(*worker).start(0xc00030c280) > >> /app/vendor/go.opencensus.io/stats/view/worker.go:276 +0x105 > >> created by go.opencensus.io/stats/view.init.0 > >> /app/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68 > >> > >> goroutine 85 [select]: > >> github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc000276780) > >> /app/pkg/logging/dedupe.go:75 +0x1e5 > >> created by github.com/prometheus/prometheus/pkg/logging.Dedupe > >> /app/pkg/logging/dedupe.go:61 +0xcf > >> > >> goroutine 92 [chan receive]: > >> > >> github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc00018ad20) > >> /app/storage/remote/write.go:96 +0xb6 > >> created by > >> github.com/prometheus/prometheus/storage/remote.NewWriteStorage > >> /app/storage/remote/write.go:89 +0x305 > >> > >> ============================================================================ > >> level=info ts=2021-01-19T05:42:16.710Z caller=main.go:360 msg="Starting > >> Prometheus" version="(version=2.23.0, branch=HEAD, > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:365 > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, > >> date=20201126-10:56:17)" > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:366 > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:367 > >> fd_limits="(soft=1048576, hard=1048576)" > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:368 > >> vm_limits="(soft=unlimited, hard=unlimited)" > >> level=info ts=2021-01-19T05:42:16.796Z caller=web.go:528 component=web > >> msg="Start listening for connections" address=0.0.0.0:9090 > >> level=info ts=2021-01-19T05:42:16.796Z caller=main.go:722 msg="Starting > >> TSDB ..." > >> level=info ts=2021-01-19T05:42:16.844Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610695556030 maxt=1610697600000 > >> ulid=01EW2S9SMSDCR35G93DT27G6NR > >> level=info ts=2021-01-19T05:42:16.875Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610697600000 maxt=1610704800000 > >> ulid=01EW2V7K5TDBRGNWYZXMMN50C7 > >> level=info ts=2021-01-19T05:42:16.901Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610704800000 maxt=1610712000000 > >> ulid=01EW323KD7FYEMJ0V03RX7TDR7 > >> level=info ts=2021-01-19T05:42:16.925Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610733600000 maxt=1610740800000 > >> ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR > >> level=info ts=2021-01-19T05:42:16.951Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610712000000 maxt=1610733600000 > >> ulid=01EW3XJEZR86TBBDJFVTBNZ8VF > >> level=info ts=2021-01-19T05:42:16.976Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610740800000 maxt=1610748000000 > >> ulid=01EW44G303K0DNBB2EDM8DX69E > >> level=info ts=2021-01-19T05:42:17.001Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610748000000 maxt=1610755200000 > >> ulid=01EW4B9NW6T794HCZATQJZJ4SZ > >> level=info ts=2021-01-19T05:42:17.049Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610755200000 maxt=1610762400000 > >> ulid=01EW4J689JDY2GWPCPDNHD0DNJ > >> level=info ts=2021-01-19T05:42:17.074Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610762400000 maxt=1610769600000 > >> ulid=01EW4S14CHH73BZHP8152M3594 > >> level=info ts=2021-01-19T05:42:17.099Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610769600000 maxt=1610776800000 > >> ulid=01EW4ZXAASHGVGEY6QZ4C6R47M > >> level=info ts=2021-01-19T05:42:17.125Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610798400000 maxt=1610805600000 > >> ulid=01EW5VBMBQBYV98Q278AHHNK70 > >> level=info ts=2021-01-19T05:42:17.155Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610776800000 maxt=1610798400000 > >> ulid=01EW5VBP3B7KSWNETV0AEKFSWY > >> level=info ts=2021-01-19T05:42:17.184Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610805600000 maxt=1610812800000 > >> ulid=01EW627M0DHYWTCC0N237SC62D > >> level=info ts=2021-01-19T05:42:17.234Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610812800000 maxt=1610820000000 > >> ulid=01EW6939M4CNXWMHGT34SD5N6R > >> level=info ts=2021-01-19T05:42:17.259Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610820000000 maxt=1610841600000 > >> ulid=01EW74JPT1T99G30CWTAE8YTVC > >> level=info ts=2021-01-19T05:42:17.285Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610863200000 maxt=1610870400000 > >> ulid=01EW7S59W18PRNC14KHMY2NQDF > >> level=info ts=2021-01-19T05:42:17.311Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610841600000 maxt=1610863200000 > >> ulid=01EW7S5CEZAKV5FHJD1BYN7VFS > >> level=info ts=2021-01-19T05:42:17.337Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610870400000 maxt=1610877600000 > >> ulid=01EW8012X70MJZYVQTEKERJSF0 > >> level=info ts=2021-01-19T05:42:17.393Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610877600000 maxt=1610884800000 > >> ulid=01EW86X36BN7KDN3YQY4KH1X7Z > >> level=info ts=2021-01-19T05:42:17.418Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610884800000 maxt=1610892000000 > >> ulid=01EW8DRFQEXQP7TCA6W9ZCWJJC > >> level=info ts=2021-01-19T05:42:17.443Z caller=repair.go:57 component=tsdb > >> msg="Found healthy block" mint=1610863200000 maxt=1610884800000 > >> ulid=01EW8DRHV7JX5CV9YKDFYBQQ99 > >> level=info ts=2021-01-19T05:42:17.606Z caller=main.go:595 msg="Stopping > >> scrape discovery manager..." > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:609 msg="Stopping > >> notify discovery manager..." > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:631 msg="Stopping > >> scrape manager..." > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:605 msg="Notify > >> discovery manager stopped" > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:591 msg="Scrape > >> discovery manager stopped" > >> level=info ts=2021-01-19T05:42:17.607Z caller=manager.go:924 > >> component="rule manager" msg="Stopping rule manager..." > >> level=info ts=2021-01-19T05:42:17.608Z caller=manager.go:934 > >> component="rule manager" msg="Rule manager stopped" > >> level=info ts=2021-01-19T05:42:17.608Z caller=notifier.go:601 > >> component=notifier msg="Stopping notification manager..." > >> level=info ts=2021-01-19T05:42:17.608Z caller=main.go:799 msg="Notifier > >> manager stopped" > >> level=info ts=2021-01-19T05:42:17.608Z caller=main.go:625 msg="Scrape > >> manager stopped" > >> level=error ts=2021-01-19T05:42:17.608Z caller=main.go:808 err="opening > >> storage failed: found unsequential head chunk files > >> /prometheus/chunks_head/000007 (index: 7) and > >> /prometheus/chunks_head/000027 (index: 27)" > >> ======================================= > >> level=info ts=2021-01-19T10:55:57.643Z caller=main.go:337 msg="Starting > >> Prometheus" version="(version=2.19.1, branch=HEAD, > >> revision=eba3fdcbf0d378b66600281903e3aab515732b39)" > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:338 > >> build_context="(go=go1.14.4, user=root@62700b3d0ef9, > >> date=20200618-16:35:26)" > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:339 > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec 10 > >> 09:13:52 UTC 2020 x86_64 prometheus-84fb48b4d9-ddq9m (none))" > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:340 > >> fd_limits="(soft=1048576, hard=1048576)" > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:341 > >> vm_limits="(soft=unlimited, hard=unlimited)" > >> level=error ts=2021-01-19T10:55:57.704Z caller=query_logger.go:93 > >> component=activeQueryTracker msg="Error setting filesize." filesize=20001 > >> err="truncate /prometheus/queries.active: interrupted system call" > >> panic: Unable to create mmap-ed active query log > >> > >> goroutine 1 [running]: > >> > >> github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff2f2aa74d, > >> > >> 0xb, 0x14, 0x2f3b7e0, 0xc00074d800, 0x2f3b7e0) > >> /app/promql/query_logger.go:117 +0x4cd > >> main.main() > >> /app/cmd/prometheus/main.go:368 +0x4ed2 > >> > >> > >> Appreciate any help > >> > >> Thanks. > >> > > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/dd2528cb-522d-44ac-900a-af2a53ad431bn%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20210222201926.GA1171695%40oxygen.
