ah, thanks for the quick reply - good spot. What would be a recommended/good/any way to store prometheus logs in some kind of storage in Azure?
On Monday, 22 February 2021 at 20:19:33 UTC Julien Pivotto wrote: > >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 > >> fs_type=fe534d42 > > That line indicates that this is using SMB - which is a network file > system not supported by Prometheus. > > > On 22 Feb 12:16, Adam Carter wrote: > > I have just setup exactly the same and find the mounting is a bit hit or > > miss - did you get anywhere with this? > > > > > > On Tuesday, 19 January 2021 at 13:23:56 UTC Sdh26 wrote: > > > > > We are not attaching Prometheus directly with NFS or SMB. We are using > > > PVC, there Kubernetes connect using protocol. > > > Container only see as regular volume attached as it is, it is > transparent > > > with Kubernetes as per my understanding. > > > On Tuesday, January 19, 2021 at 5:39:43 PM UTC+5:30 Sdh26 wrote: > > > > > >> Hello , > > >> > > >> We are facing an issue with Prometheus which is running on Azure > > >> Kubernetes Service (v1.19.1). Nodes are Ubuntu 18.04. > > >> We are using Azure Storage File Share to store the prometheus data. > > >> The Prometheus pods runs for couple of hours if we clear the data but > > >> then keeps going to CrashLoopBackOff and throws multiple errors. > > >> These are the Logs which we have gathered on multiple instances. We > were > > >> using Prometheus v2.19.1 but we tried with Prometheus version 2.23.0 > also > > >> but doesn't improve. > > >> The issue started since we upgraded the Kubernetes version from > 1.17.9 to > > >> 1.19.1. > > >> Is there any compatibility issue during upgrade? > > >> =============================================== > > >> > > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:360 > msg="Starting > > >> Prometheus" version="(version=2.23.0, branch=HEAD, > > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:365 > > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, > > >> date=20201126-10:56:17)" > > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:366 > > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec > 10 > > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:367 > > >> fd_limits="(soft=1048576, hard=1048576)" > > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:368 > > >> vm_limits="(soft=unlimited, hard=unlimited)" > > >> level=info ts=2021-01-15T07:25:31.526Z caller=main.go:722 > msg="Starting > > >> TSDB ..." > > >> level=info ts=2021-01-15T07:25:31.526Z caller=web.go:528 > component=web > > >> msg="Start listening for connections" address=0.0.0.0:9090 > > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:645 > component=tsdb > > >> msg="Replaying on-disk memory mappable chunks if any" > > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:659 > component=tsdb > > >> msg="On-disk memory mappable chunks replay completed" duration=4.5盜 > > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:665 > component=tsdb > > >> msg="Replaying WAL, this may take a while" > > >> level=info ts=2021-01-15T07:25:31.824Z caller=head.go:717 > component=tsdb > > >> msg="WAL segment loaded" segment=0 maxSegment=0 > > >> level=info ts=2021-01-15T07:25:31.824Z caller=head.go:722 > component=tsdb > > >> msg="WAL replay completed" checkpoint_replay_duration=23.473299ms > > >> wal_replay_duration=43.082181ms total_replay_duration=66.57608ms > > >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 > fs_type=fe534d42 > > >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:745 msg="TSDB > > >> started" > > >> level=info ts=2021-01-15T07:25:31.862Z caller=main.go:871 > msg="Loading > > >> configuration file" filename=/etc/prometheus/prometheus.yml > > >> level=info ts=2021-01-15T07:25:31.950Z caller=main.go:902 > msg="Completed > > >> loading of configuration file" > filename=/etc/prometheus/prometheus.yml > > >> totalDuration=88.228971ms remote_storage=2.901盜 web_handler=500ns > > >> query_engine=1.2盜 scrape=380.501盜 scrape_sd=135.301盜 notify=27.1盜 > > >> notify_sd=14.6盜 rules=67.262483ms > > >> level=info ts=2021-01-15T07:25:31.950Z caller=main.go:694 msg="Server > is > > >> ready to receive web requests." > > >> level=error ts=2021-01-15T07:52:33.026Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T08:18:36.113Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T08:21:38.206Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T08:24:40.278Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T08:40:42.841Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T09:50:47.814Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T10:26:03.298Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: > populate > > >> block: write chunks: preallocate: seek > > >> > /prometheus/01EW2S98EFR81VFT0677NH81ZF.tmp-for-creation/chunks/000001.tmp: > > >> interrupted system call" > > >> level=error ts=2021-01-15T10:26:05.816Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: > populate > > >> block: write chunks: preallocate: seek > > >> > /prometheus/01EW2S9AXV1E42NHPGV0K9Z2MC.tmp-for-creation/chunks/000001.tmp: > > >> interrupted system call" > > >> level=error ts=2021-01-15T10:26:15.075Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: > populate > > >> block: write chunks: preallocate: seek > > >> > /prometheus/01EW2S9KZ7GZZ1N36NFMNDCKRV.tmp-for-creation/chunks/000001.tmp: > > >> interrupted system call" > > >> level=info ts=2021-01-15T10:26:21.209Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610695556030 > maxt=1610697600000 > > >> ulid=01EW2S9SMSDCR35G93DT27G6NR duration=767.820675ms > > >> level=info ts=2021-01-15T10:26:21.412Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=25.718613ms > > >> level=error ts=2021-01-15T10:36:22.279Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T10:55:25.565Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T10:56:27.577Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=info ts=2021-01-15T11:00:06.243Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610697600000 > maxt=1610704800000 > > >> ulid=01EW2V7K5TDBRGNWYZXMMN50C7 duration=808.923904ms > > >> level=info ts=2021-01-15T11:00:06.487Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=33.110139ms > > >> level=error ts=2021-01-15T11:48:11.574Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T11:55:14.296Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T12:09:17.495Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T12:24:20.974Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T12:51:25.739Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T12:55:28.056Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T13:00:02.893Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: open > chunk > > >> writer: mkdir > > >> /prometheus/01EW3237WF9WWH4TDY3HA2KBRK.tmp-for-creation/chunks: > interrupted > > >> system call" > > >> level=error ts=2021-01-15T13:00:05.471Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: open > chunk > > >> writer: mkdir > > >> /prometheus/01EW323AD1YAQ6VXB2G6Q6HZ58.tmp-for-creation/chunks: > interrupted > > >> system call" > > >> level=info ts=2021-01-15T13:00:15.444Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610704800000 > maxt=1610712000000 > > >> ulid=01EW323KD7FYEMJ0V03RX7TDR7 duration=812.389067ms > > >> level=info ts=2021-01-15T13:00:15.725Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=31.144829ms > > >> level=error ts=2021-01-15T13:07:16.892Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T13:35:22.717Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T13:42:25.587Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T13:47:28.162Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T13:55:31.084Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T14:19:36.277Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T14:24:38.878Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T14:54:45.004Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T14:56:47.144Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T15:00:06.054Z caller=compact.go:541 > > >> component=tsdb msg="removed tmp folder after failed compaction" > > >> err="unlinkat > /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation: > > >> directory not empty" > > >> level=error ts=2021-01-15T15:00:06.054Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: sync > > >> /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation/index: > interrupted > > >> system call" > > >> level=info ts=2021-01-15T15:00:07.791Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610712000000 > maxt=1610719200000 > > >> ulid=01EW38Z38E2TA8CDEJNJ130QV9 duration=736.679699ms > > >> level=info ts=2021-01-15T15:00:08.081Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=27.759017ms > > >> level=info ts=2021-01-15T15:00:08.261Z caller=checkpoint.go:95 > > >> component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 > > >> mint=1610719200000 > > >> level=info ts=2021-01-15T15:00:08.689Z caller=head.go:908 > component=tsdb > > >> msg="WAL checkpoint complete" first=0 last=1 duration=454.463412ms > > >> level=error ts=2021-01-15T15:01:08.914Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T15:17:13.515Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T15:44:19.858Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T15:50:22.808Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T15:54:25.360Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T15:55:27.371Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T16:02:32.493Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T16:09:35.501Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T16:17:38.646Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T16:19:40.796Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T16:28:44.071Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=info ts=2021-01-15T17:00:01.827Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610719200000 > maxt=1610726400000 > > >> ulid=01EW3FTMKQYF12V439SBT6B17P duration=811.71891ms > > >> level=info ts=2021-01-15T17:00:02.175Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=28.805321ms > > >> level=error ts=2021-01-15T17:00:03.791Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact > > >> [/prometheus/01EW2S9SMSDCR35G93DT27G6NR > > >> /prometheus/01EW2V7K5TDBRGNWYZXMMN50C7 > > >> /prometheus/01EW323KD7FYEMJ0V03RX7TDR7]: write merged meta: rename > > >> /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json.tmp > > >> /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json: > > >> interrupted system call" > > >> level=error ts=2021-01-15T17:01:05.830Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T17:03:10.067Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T17:18:14.829Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T17:24:17.637Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T17:36:21.558Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T17:38:23.743Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T17:48:27.143Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T18:12:32.875Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T18:20:36.002Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T18:24:38.461Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T18:32:41.690Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T18:36:44.188Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T18:42:47.011Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T19:00:06.007Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: write > merged > > >> meta: rename > > >> /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json.tmp > > >> /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json: > > >> interrupted system call" > > >> level=info ts=2021-01-15T19:00:07.649Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610726400000 > maxt=1610733600000 > > >> ulid=01EW3PPHPZ6ZZM2JGKF69F96RN duration=641.827141ms > > >> level=info ts=2021-01-15T19:00:07.932Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=24.2159ms > > >> level=info ts=2021-01-15T19:00:08.096Z caller=checkpoint.go:95 > > >> component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 > > >> mint=1610733600000 > > >> level=info ts=2021-01-15T19:00:08.589Z caller=head.go:908 > component=tsdb > > >> msg="WAL checkpoint complete" first=2 last=3 duration=516.496925ms > > >> level=error ts=2021-01-15T19:08:09.957Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T19:16:13.284Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T19:19:15.693Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T19:20:17.699Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T19:39:24.830Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T19:42:27.215Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T19:45:29.593Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T19:56:33.420Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:06:37.228Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:08:39.420Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:19:43.242Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:30:47.040Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:35:49.798Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:38:52.166Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:49:55.966Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:50:57.973Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T20:59:03.232Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T21:00:05.262Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T21:00:07.878Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: write > merged > > >> meta: rename > > >> /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json.tmp > > >> /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json: > > >> interrupted system call" > > >> level=info ts=2021-01-15T21:00:12.523Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610733600000 > maxt=1610740800000 > > >> ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR duration=644.683302ms > > >> level=info ts=2021-01-15T21:00:12.828Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=22.771295ms > > >> level=info ts=2021-01-15T21:00:13.960Z caller=compact.go:443 > > >> component=tsdb msg="compact blocks" count=3 mint=1610712000000 > > >> maxt=1610733600000 ulid=01EW3XJEZR86TBBDJFVTBNZ8VF > > >> sources="[01EW38Z38E2TA8CDEJNJ130QV9 01EW3FTMKQYF12V439SBT6B17P > > >> 01EW3PPHPZ6ZZM2JGKF69F96RN]" duration=827.624569ms > > >> level=info ts=2021-01-15T21:00:14.381Z caller=db.go:1187 > component=tsdb > > >> msg="Deleting obsolete block" block=01EW38Z38E2TA8CDEJNJ130QV9 > > >> level=info ts=2021-01-15T21:00:14.547Z caller=db.go:1187 > component=tsdb > > >> msg="Deleting obsolete block" block=01EW3FTMKQYF12V439SBT6B17P > > >> level=info ts=2021-01-15T21:00:14.705Z caller=db.go:1187 > component=tsdb > > >> msg="Deleting obsolete block" block=01EW3PPHPZ6ZZM2JGKF69F96RN > > >> level=error ts=2021-01-15T21:03:15.152Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T21:20:19.710Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T21:36:24.045Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T21:38:26.227Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T21:46:29.339Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T21:47:31.378Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T22:00:37.302Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T22:02:39.487Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T22:17:43.600Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T22:26:47.082Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T22:35:50.353Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T22:36:52.383Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T23:00:06.094Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: write > new > > >> tombstones file: rename > > >> > /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones.tmp > > >> /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones: > > >> interrupted system call" > > >> level=error ts=2021-01-15T23:00:07.633Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: write > merged > > >> meta: rename > > >> /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json.tmp > > >> /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json: > > >> interrupted system call" > > >> level=error ts=2021-01-15T23:00:13.695Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: rename > block > > >> dir: rename /prometheus/01EW44E60NC3EGCETC25GBW492.tmp-for-creation > > >> /prometheus/01EW44E60NC3EGCETC25GBW492: interrupted system call" > > >> level=error ts=2021-01-15T23:00:18.059Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: > populate > > >> block: write chunks: preallocate: seek > > >> > /prometheus/01EW44EAMZKJ5JJ2E42WCM8V0Z.tmp-for-creation/chunks/000001.tmp: > > >> interrupted system call" > > >> level=error ts=2021-01-15T23:00:26.689Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: write > merged > > >> meta: rename > > >> /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json.tmp > > >> /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json: > > >> interrupted system call" > > >> level=error ts=2021-01-15T23:00:43.395Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: write > new > > >> tombstones file: rename > > >> > /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones.tmp > > >> /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones: > > >> interrupted system call" > > >> level=info ts=2021-01-15T23:01:16.055Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610740800000 > maxt=1610748000000 > > >> ulid=01EW44G303K0DNBB2EDM8DX69E duration=659.365701ms > > >> level=info ts=2021-01-15T23:01:16.353Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=23.148498ms > > >> level=info ts=2021-01-15T23:01:16.514Z caller=checkpoint.go:95 > > >> component=tsdb msg="Creating checkpoint" from_segment=4 to_segment=5 > > >> mint=1610748000000 > > >> level=info ts=2021-01-15T23:01:17.004Z caller=head.go:908 > component=tsdb > > >> msg="WAL checkpoint complete" first=4 last=5 duration=512.755478ms > > >> level=error ts=2021-01-15T23:22:20.652Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T23:25:23.109Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T23:40:27.567Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-15T23:50:31.230Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T00:03:35.502Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T00:20:40.429Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T00:21:42.458Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T00:23:46.659Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T00:25:48.862Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T00:37:52.910Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T00:57:58.400Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T01:00:00.602Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=info ts=2021-01-16T01:00:06.068Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610748000000 > maxt=1610755200000 > > >> ulid=01EW4B9NW6T794HCZATQJZJ4SZ duration=686.024281ms > > >> level=info ts=2021-01-16T01:00:06.429Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=22.475094ms > > >> level=error ts=2021-01-16T01:00:06.563Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: head memory truncate: > truncate > > >> chunks.HeadReadWriter: remove /prometheus/chunks_head/000007: > interrupted > > >> system call" > > >> level=error ts=2021-01-16T01:08:12.075Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T01:15:15.300Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T01:20:18.133Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T01:22:20.357Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T01:31:24.051Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T01:50:29.872Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T01:52:32.155Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T02:00:35.607Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T02:40:45.873Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T02:54:50.472Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="plan compaction: readdirent: no such > file or > > >> directory" > > >> level=error ts=2021-01-16T03:00:06.014Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: write > merged > > >> meta: rename > > >> /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json.tmp > > >> /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json: > > >> interrupted system call" > > >> level=error ts=2021-01-16T03:00:16.046Z caller=compact.go:541 > > >> component=tsdb msg="removed tmp folder after failed compaction" > > >> err="unlinkat > /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation: > > >> directory not empty" > > >> level=error ts=2021-01-16T03:00:16.046Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: sync > > >> /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation/index: > interrupted > > >> system call" > > >> level=error ts=2021-01-16T03:00:20.488Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: open > chunk > > >> writer: mkdir > > >> /prometheus/01EW4J5VTSRT3KXABDFXETHK99.tmp-for-creation/chunks: > interrupted > > >> system call" > > >> level=error ts=2021-01-16T03:00:25.202Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact head: persist head block: write > new > > >> tombstones file: rename > > >> > /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones.tmp > > >> /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones: > > >> interrupted system call" > > >> level=info ts=2021-01-16T03:00:33.853Z caller=compact.go:502 > > >> component=tsdb msg="write block" mint=1610755200000 > maxt=1610762400000 > > >> ulid=01EW4J689JDY2GWPCPDNHD0DNJ duration=650.542029ms > > >> level=info ts=2021-01-16T03:00:34.211Z caller=head.go:824 > component=tsdb > > >> msg="Head GC completed" duration=22.278794ms > > >> level=error ts=2021-01-16T03:00:35.654Z caller=db.go:746 > component=tsdb > > >> msg="compaction failed" err="compact > > >> [/prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR > > >> /prometheus/01EW44G303K0DNBB2EDM8DX69E > > >> /prometheus/01EW4B9NW6T794HCZATQJZJ4SZ]: 2 errors: rename block dir: > rename > > >> /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK.tmp-for-creation > > >> /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK: interrupted system call; > setting > > >> compaction failed for block: /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR: > rename > > >> /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json.tmp > > >> /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json: interrupted system > call" > > >> > > >> > > >> =============================== > > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:360 > msg="Starting > > >> Prometheus" version="(version=2.23.0, branch=HEAD, > > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:365 > > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, > > >> date=20201126-10:56:17)" > > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:366 > > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec > 10 > > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:367 > > >> fd_limits="(soft=1048576, hard=1048576)" > > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:368 > > >> vm_limits="(soft=unlimited, hard=unlimited)" > > >> level=error ts=2021-01-19T05:06:08.726Z caller=query_logger.go:71 > > >> component=activeQueryTracker msg="Failed to read query log file" > err=EOF > > >> unexpected fault address 0x7f3c06738000 > > >> fatal error: fault > > >> [signal SIGBUS: bus error code=0x2 addr=0x7f3c06738000 pc=0x470ea8] > > >> > > >> goroutine 1 [running]: > > >> runtime.throw(0x29f4a25, 0x5) > > >> /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00092b248 > > >> sp=0xc00092b218 pc=0x4378b2 > > >> runtime.sigpanic() > > >> /usr/local/go/src/runtime/signal_unix.go:739 +0x485 fp=0xc00092b278 > > >> sp=0xc00092b248 pc=0x44e085 > > >> runtime.memmove(0x7f3c06738000, 0x29f1e48, 0x1) > > >> /usr/local/go/src/runtime/memmove_amd64.s:148 +0x108 fp=0xc00092b280 > > >> sp=0xc00092b278 pc=0x470ea8 > > >> > > >> > github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff842a774d, > > > >> 0xb, 0x14, 0x30d9a40, 0xc00077ede0, 0x30d9a40) > > >> /app/promql/query_logger.go:120 +0x3a5 fp=0xc00092b3d8 > sp=0xc00092b280 > > >> pc=0x20fac05 > > >> main.main() > > >> /app/cmd/prometheus/main.go:396 +0x53cc fp=0xc00092bf88 > sp=0xc00092b3d8 > > >> pc=0x21f9a0c > > >> runtime.main() > > >> /usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc00092bfe0 > > >> sp=0xc00092bf88 pc=0x43a0a9 > > >> runtime.goexit() > > >> /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00092bfe8 > > >> sp=0xc00092bfe0 pc=0x46ffa1 > > >> > > >> goroutine 9 [select]: > > >> go.opencensus.io/stats/view.(*worker).start(0xc00030c280) > > >> /app/vendor/go.opencensus.io/stats/view/worker.go:276 +0x105 > > >> created by go.opencensus.io/stats/view.init.0 > > >> /app/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68 > > >> > > >> goroutine 85 [select]: > > >> > github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc000276780) > > >> /app/pkg/logging/dedupe.go:75 +0x1e5 > > >> created by github.com/prometheus/prometheus/pkg/logging.Dedupe > > >> /app/pkg/logging/dedupe.go:61 +0xcf > > >> > > >> goroutine 92 [chan receive]: > > >> > > >> > github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc00018ad20) > > >> /app/storage/remote/write.go:96 +0xb6 > > >> created by > > >> github.com/prometheus/prometheus/storage/remote.NewWriteStorage > > >> /app/storage/remote/write.go:89 +0x305 > > >> > > >> > ============================================================================ > > >> level=info ts=2021-01-19T05:42:16.710Z caller=main.go:360 > msg="Starting > > >> Prometheus" version="(version=2.23.0, branch=HEAD, > > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" > > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:365 > > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, > > >> date=20201126-10:56:17)" > > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:366 > > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec > 10 > > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" > > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:367 > > >> fd_limits="(soft=1048576, hard=1048576)" > > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:368 > > >> vm_limits="(soft=unlimited, hard=unlimited)" > > >> level=info ts=2021-01-19T05:42:16.796Z caller=web.go:528 > component=web > > >> msg="Start listening for connections" address=0.0.0.0:9090 > > >> level=info ts=2021-01-19T05:42:16.796Z caller=main.go:722 > msg="Starting > > >> TSDB ..." > > >> level=info ts=2021-01-19T05:42:16.844Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610695556030 maxt=1610697600000 > > >> ulid=01EW2S9SMSDCR35G93DT27G6NR > > >> level=info ts=2021-01-19T05:42:16.875Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610697600000 maxt=1610704800000 > > >> ulid=01EW2V7K5TDBRGNWYZXMMN50C7 > > >> level=info ts=2021-01-19T05:42:16.901Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610704800000 maxt=1610712000000 > > >> ulid=01EW323KD7FYEMJ0V03RX7TDR7 > > >> level=info ts=2021-01-19T05:42:16.925Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610733600000 maxt=1610740800000 > > >> ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR > > >> level=info ts=2021-01-19T05:42:16.951Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610712000000 maxt=1610733600000 > > >> ulid=01EW3XJEZR86TBBDJFVTBNZ8VF > > >> level=info ts=2021-01-19T05:42:16.976Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610740800000 maxt=1610748000000 > > >> ulid=01EW44G303K0DNBB2EDM8DX69E > > >> level=info ts=2021-01-19T05:42:17.001Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610748000000 maxt=1610755200000 > > >> ulid=01EW4B9NW6T794HCZATQJZJ4SZ > > >> level=info ts=2021-01-19T05:42:17.049Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610755200000 maxt=1610762400000 > > >> ulid=01EW4J689JDY2GWPCPDNHD0DNJ > > >> level=info ts=2021-01-19T05:42:17.074Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610762400000 maxt=1610769600000 > > >> ulid=01EW4S14CHH73BZHP8152M3594 > > >> level=info ts=2021-01-19T05:42:17.099Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610769600000 maxt=1610776800000 > > >> ulid=01EW4ZXAASHGVGEY6QZ4C6R47M > > >> level=info ts=2021-01-19T05:42:17.125Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610798400000 maxt=1610805600000 > > >> ulid=01EW5VBMBQBYV98Q278AHHNK70 > > >> level=info ts=2021-01-19T05:42:17.155Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610776800000 maxt=1610798400000 > > >> ulid=01EW5VBP3B7KSWNETV0AEKFSWY > > >> level=info ts=2021-01-19T05:42:17.184Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610805600000 maxt=1610812800000 > > >> ulid=01EW627M0DHYWTCC0N237SC62D > > >> level=info ts=2021-01-19T05:42:17.234Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610812800000 maxt=1610820000000 > > >> ulid=01EW6939M4CNXWMHGT34SD5N6R > > >> level=info ts=2021-01-19T05:42:17.259Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610820000000 maxt=1610841600000 > > >> ulid=01EW74JPT1T99G30CWTAE8YTVC > > >> level=info ts=2021-01-19T05:42:17.285Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610863200000 maxt=1610870400000 > > >> ulid=01EW7S59W18PRNC14KHMY2NQDF > > >> level=info ts=2021-01-19T05:42:17.311Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610841600000 maxt=1610863200000 > > >> ulid=01EW7S5CEZAKV5FHJD1BYN7VFS > > >> level=info ts=2021-01-19T05:42:17.337Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610870400000 maxt=1610877600000 > > >> ulid=01EW8012X70MJZYVQTEKERJSF0 > > >> level=info ts=2021-01-19T05:42:17.393Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610877600000 maxt=1610884800000 > > >> ulid=01EW86X36BN7KDN3YQY4KH1X7Z > > >> level=info ts=2021-01-19T05:42:17.418Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610884800000 maxt=1610892000000 > > >> ulid=01EW8DRFQEXQP7TCA6W9ZCWJJC > > >> level=info ts=2021-01-19T05:42:17.443Z caller=repair.go:57 > component=tsdb > > >> msg="Found healthy block" mint=1610863200000 maxt=1610884800000 > > >> ulid=01EW8DRHV7JX5CV9YKDFYBQQ99 > > >> level=info ts=2021-01-19T05:42:17.606Z caller=main.go:595 > msg="Stopping > > >> scrape discovery manager..." > > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:609 > msg="Stopping > > >> notify discovery manager..." > > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:631 > msg="Stopping > > >> scrape manager..." > > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:605 msg="Notify > > >> discovery manager stopped" > > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:591 msg="Scrape > > >> discovery manager stopped" > > >> level=info ts=2021-01-19T05:42:17.607Z caller=manager.go:924 > > >> component="rule manager" msg="Stopping rule manager..." > > >> level=info ts=2021-01-19T05:42:17.608Z caller=manager.go:934 > > >> component="rule manager" msg="Rule manager stopped" > > >> level=info ts=2021-01-19T05:42:17.608Z caller=notifier.go:601 > > >> component=notifier msg="Stopping notification manager..." > > >> level=info ts=2021-01-19T05:42:17.608Z caller=main.go:799 > msg="Notifier > > >> manager stopped" > > >> level=info ts=2021-01-19T05:42:17.608Z caller=main.go:625 msg="Scrape > > >> manager stopped" > > >> level=error ts=2021-01-19T05:42:17.608Z caller=main.go:808 > err="opening > > >> storage failed: found unsequential head chunk files > > >> /prometheus/chunks_head/000007 (index: 7) and > > >> /prometheus/chunks_head/000027 (index: 27)" > > >> ======================================= > > >> level=info ts=2021-01-19T10:55:57.643Z caller=main.go:337 > msg="Starting > > >> Prometheus" version="(version=2.19.1, branch=HEAD, > > >> revision=eba3fdcbf0d378b66600281903e3aab515732b39)" > > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:338 > > >> build_context="(go=go1.14.4, user=root@62700b3d0ef9, > > >> date=20200618-16:35:26)" > > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:339 > > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec > 10 > > >> 09:13:52 UTC 2020 x86_64 prometheus-84fb48b4d9-ddq9m (none))" > > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:340 > > >> fd_limits="(soft=1048576, hard=1048576)" > > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:341 > > >> vm_limits="(soft=unlimited, hard=unlimited)" > > >> level=error ts=2021-01-19T10:55:57.704Z caller=query_logger.go:93 > > >> component=activeQueryTracker msg="Error setting filesize." > filesize=20001 > > >> err="truncate /prometheus/queries.active: interrupted system call" > > >> panic: Unable to create mmap-ed active query log > > >> > > >> goroutine 1 [running]: > > >> > > >> > github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff2f2aa74d, > > > >> 0xb, 0x14, 0x2f3b7e0, 0xc00074d800, 0x2f3b7e0) > > >> /app/promql/query_logger.go:117 +0x4cd > > >> main.main() > > >> /app/cmd/prometheus/main.go:368 +0x4ed2 > > >> > > >> > > >> Appreciate any help > > >> > > >> Thanks. > > >> > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/dd2528cb-522d-44ac-900a-af2a53ad431bn%40googlegroups.com > . > > > -- > Julien Pivotto > @roidelapluie > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b741a489-8905-4abd-bcb6-96c2c18b8845n%40googlegroups.com.
