Microsoft suggested to use Azure Disk On Tuesday, February 23, 2021 at 2:06:13 AM UTC+5:30 [email protected] wrote:
> ah, thanks for the quick reply - good spot. What would be a > recommended/good/any way to store prometheus logs in some kind of storage > in Azure? > > On Monday, 22 February 2021 at 20:19:33 UTC Julien Pivotto wrote: > >> >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 >> >> fs_type=fe534d42 >> >> That line indicates that this is using SMB - which is a network file >> system not supported by Prometheus. >> >> >> On 22 Feb 12:16, Adam Carter wrote: >> > I have just setup exactly the same and find the mounting is a bit hit >> or >> > miss - did you get anywhere with this? >> > >> > >> > On Tuesday, 19 January 2021 at 13:23:56 UTC Sdh26 wrote: >> > >> > > We are not attaching Prometheus directly with NFS or SMB. We are >> using >> > > PVC, there Kubernetes connect using protocol. >> > > Container only see as regular volume attached as it is, it is >> transparent >> > > with Kubernetes as per my understanding. >> > > On Tuesday, January 19, 2021 at 5:39:43 PM UTC+5:30 Sdh26 wrote: >> > > >> > >> Hello , >> > >> >> > >> We are facing an issue with Prometheus which is running on Azure >> > >> Kubernetes Service (v1.19.1). Nodes are Ubuntu 18.04. >> > >> We are using Azure Storage File Share to store the prometheus data. >> > >> The Prometheus pods runs for couple of hours if we clear the data >> but >> > >> then keeps going to CrashLoopBackOff and throws multiple errors. >> > >> These are the Logs which we have gathered on multiple instances. We >> were >> > >> using Prometheus v2.19.1 but we tried with Prometheus version 2.23.0 >> also >> > >> but doesn't improve. >> > >> The issue started since we upgraded the Kubernetes version from >> 1.17.9 to >> > >> 1.19.1. >> > >> Is there any compatibility issue during upgrade? >> > >> =============================================== >> > >> >> > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:360 >> msg="Starting >> > >> Prometheus" version="(version=2.23.0, branch=HEAD, >> > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" >> > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:365 >> > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, >> > >> date=20201126-10:56:17)" >> > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:366 >> > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec >> 10 >> > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" >> > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:367 >> > >> fd_limits="(soft=1048576, hard=1048576)" >> > >> level=info ts=2021-01-15T07:25:31.426Z caller=main.go:368 >> > >> vm_limits="(soft=unlimited, hard=unlimited)" >> > >> level=info ts=2021-01-15T07:25:31.526Z caller=main.go:722 >> msg="Starting >> > >> TSDB ..." >> > >> level=info ts=2021-01-15T07:25:31.526Z caller=web.go:528 >> component=web >> > >> msg="Start listening for connections" address=0.0.0.0:9090 >> > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:645 >> component=tsdb >> > >> msg="Replaying on-disk memory mappable chunks if any" >> > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:659 >> component=tsdb >> > >> msg="On-disk memory mappable chunks replay completed" duration=4.5盜 >> > >> level=info ts=2021-01-15T07:25:31.757Z caller=head.go:665 >> component=tsdb >> > >> msg="Replaying WAL, this may take a while" >> > >> level=info ts=2021-01-15T07:25:31.824Z caller=head.go:717 >> component=tsdb >> > >> msg="WAL segment loaded" segment=0 maxSegment=0 >> > >> level=info ts=2021-01-15T07:25:31.824Z caller=head.go:722 >> component=tsdb >> > >> msg="WAL replay completed" checkpoint_replay_duration=23.473299ms >> > >> wal_replay_duration=43.082181ms total_replay_duration=66.57608ms >> > >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:742 >> fs_type=fe534d42 >> > >> level=info ts=2021-01-15T07:25:31.861Z caller=main.go:745 msg="TSDB >> > >> started" >> > >> level=info ts=2021-01-15T07:25:31.862Z caller=main.go:871 >> msg="Loading >> > >> configuration file" filename=/etc/prometheus/prometheus.yml >> > >> level=info ts=2021-01-15T07:25:31.950Z caller=main.go:902 >> msg="Completed >> > >> loading of configuration file" >> filename=/etc/prometheus/prometheus.yml >> > >> totalDuration=88.228971ms remote_storage=2.901盜 web_handler=500ns >> > >> query_engine=1.2盜 scrape=380.501盜 scrape_sd=135.301盜 notify=27.1盜 >> > >> notify_sd=14.6盜 rules=67.262483ms >> > >> level=info ts=2021-01-15T07:25:31.950Z caller=main.go:694 >> msg="Server is >> > >> ready to receive web requests." >> > >> level=error ts=2021-01-15T07:52:33.026Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T08:18:36.113Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T08:21:38.206Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T08:24:40.278Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T08:40:42.841Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T09:50:47.814Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T10:26:03.298Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: >> populate >> > >> block: write chunks: preallocate: seek >> > >> >> /prometheus/01EW2S98EFR81VFT0677NH81ZF.tmp-for-creation/chunks/000001.tmp: >> > >> interrupted system call" >> > >> level=error ts=2021-01-15T10:26:05.816Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: >> populate >> > >> block: write chunks: preallocate: seek >> > >> >> /prometheus/01EW2S9AXV1E42NHPGV0K9Z2MC.tmp-for-creation/chunks/000001.tmp: >> > >> interrupted system call" >> > >> level=error ts=2021-01-15T10:26:15.075Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: >> populate >> > >> block: write chunks: preallocate: seek >> > >> >> /prometheus/01EW2S9KZ7GZZ1N36NFMNDCKRV.tmp-for-creation/chunks/000001.tmp: >> > >> interrupted system call" >> > >> level=info ts=2021-01-15T10:26:21.209Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610695556030 >> maxt=1610697600000 >> > >> ulid=01EW2S9SMSDCR35G93DT27G6NR duration=767.820675ms >> > >> level=info ts=2021-01-15T10:26:21.412Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=25.718613ms >> > >> level=error ts=2021-01-15T10:36:22.279Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T10:55:25.565Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T10:56:27.577Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=info ts=2021-01-15T11:00:06.243Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610697600000 >> maxt=1610704800000 >> > >> ulid=01EW2V7K5TDBRGNWYZXMMN50C7 duration=808.923904ms >> > >> level=info ts=2021-01-15T11:00:06.487Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=33.110139ms >> > >> level=error ts=2021-01-15T11:48:11.574Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T11:55:14.296Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T12:09:17.495Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T12:24:20.974Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T12:51:25.739Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T12:55:28.056Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T13:00:02.893Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: open >> chunk >> > >> writer: mkdir >> > >> /prometheus/01EW3237WF9WWH4TDY3HA2KBRK.tmp-for-creation/chunks: >> interrupted >> > >> system call" >> > >> level=error ts=2021-01-15T13:00:05.471Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: open >> chunk >> > >> writer: mkdir >> > >> /prometheus/01EW323AD1YAQ6VXB2G6Q6HZ58.tmp-for-creation/chunks: >> interrupted >> > >> system call" >> > >> level=info ts=2021-01-15T13:00:15.444Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610704800000 >> maxt=1610712000000 >> > >> ulid=01EW323KD7FYEMJ0V03RX7TDR7 duration=812.389067ms >> > >> level=info ts=2021-01-15T13:00:15.725Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=31.144829ms >> > >> level=error ts=2021-01-15T13:07:16.892Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T13:35:22.717Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T13:42:25.587Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T13:47:28.162Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T13:55:31.084Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T14:19:36.277Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T14:24:38.878Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T14:54:45.004Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T14:56:47.144Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T15:00:06.054Z caller=compact.go:541 >> > >> component=tsdb msg="removed tmp folder after failed compaction" >> > >> err="unlinkat >> /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation: >> > >> directory not empty" >> > >> level=error ts=2021-01-15T15:00:06.054Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: sync >> > >> /prometheus/01EW38Z1NADD1VD2XXD56RHRTN.tmp-for-creation/index: >> interrupted >> > >> system call" >> > >> level=info ts=2021-01-15T15:00:07.791Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610712000000 >> maxt=1610719200000 >> > >> ulid=01EW38Z38E2TA8CDEJNJ130QV9 duration=736.679699ms >> > >> level=info ts=2021-01-15T15:00:08.081Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=27.759017ms >> > >> level=info ts=2021-01-15T15:00:08.261Z caller=checkpoint.go:95 >> > >> component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 >> > >> mint=1610719200000 >> > >> level=info ts=2021-01-15T15:00:08.689Z caller=head.go:908 >> component=tsdb >> > >> msg="WAL checkpoint complete" first=0 last=1 duration=454.463412ms >> > >> level=error ts=2021-01-15T15:01:08.914Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T15:17:13.515Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T15:44:19.858Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T15:50:22.808Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T15:54:25.360Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T15:55:27.371Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T16:02:32.493Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T16:09:35.501Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T16:17:38.646Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T16:19:40.796Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T16:28:44.071Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=info ts=2021-01-15T17:00:01.827Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610719200000 >> maxt=1610726400000 >> > >> ulid=01EW3FTMKQYF12V439SBT6B17P duration=811.71891ms >> > >> level=info ts=2021-01-15T17:00:02.175Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=28.805321ms >> > >> level=error ts=2021-01-15T17:00:03.791Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact >> > >> [/prometheus/01EW2S9SMSDCR35G93DT27G6NR >> > >> /prometheus/01EW2V7K5TDBRGNWYZXMMN50C7 >> > >> /prometheus/01EW323KD7FYEMJ0V03RX7TDR7]: write merged meta: rename >> > >> >> /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json.tmp >> > >> /prometheus/01EW3FTP5AB3NAWXTNSJ0CX7KV.tmp-for-creation/meta.json: >> > >> interrupted system call" >> > >> level=error ts=2021-01-15T17:01:05.830Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T17:03:10.067Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T17:18:14.829Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T17:24:17.637Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T17:36:21.558Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T17:38:23.743Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T17:48:27.143Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T18:12:32.875Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T18:20:36.002Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T18:24:38.461Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T18:32:41.690Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T18:36:44.188Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T18:42:47.011Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T19:00:06.007Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: write >> merged >> > >> meta: rename >> > >> >> /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json.tmp >> > >> /prometheus/01EW3PPG44R4430NYXX0WT7330.tmp-for-creation/meta.json: >> > >> interrupted system call" >> > >> level=info ts=2021-01-15T19:00:07.649Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610726400000 >> maxt=1610733600000 >> > >> ulid=01EW3PPHPZ6ZZM2JGKF69F96RN duration=641.827141ms >> > >> level=info ts=2021-01-15T19:00:07.932Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=24.2159ms >> > >> level=info ts=2021-01-15T19:00:08.096Z caller=checkpoint.go:95 >> > >> component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 >> > >> mint=1610733600000 >> > >> level=info ts=2021-01-15T19:00:08.589Z caller=head.go:908 >> component=tsdb >> > >> msg="WAL checkpoint complete" first=2 last=3 duration=516.496925ms >> > >> level=error ts=2021-01-15T19:08:09.957Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T19:16:13.284Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T19:19:15.693Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T19:20:17.699Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T19:39:24.830Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T19:42:27.215Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T19:45:29.593Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T19:56:33.420Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:06:37.228Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:08:39.420Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:19:43.242Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:30:47.040Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:35:49.798Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:38:52.166Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:49:55.966Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:50:57.973Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T20:59:03.232Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T21:00:05.262Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T21:00:07.878Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: write >> merged >> > >> meta: rename >> > >> >> /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json.tmp >> > >> /prometheus/01EW3XJ96YW55V4NB3PPQKK93V.tmp-for-creation/meta.json: >> > >> interrupted system call" >> > >> level=info ts=2021-01-15T21:00:12.523Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610733600000 >> maxt=1610740800000 >> > >> ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR duration=644.683302ms >> > >> level=info ts=2021-01-15T21:00:12.828Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=22.771295ms >> > >> level=info ts=2021-01-15T21:00:13.960Z caller=compact.go:443 >> > >> component=tsdb msg="compact blocks" count=3 mint=1610712000000 >> > >> maxt=1610733600000 ulid=01EW3XJEZR86TBBDJFVTBNZ8VF >> > >> sources="[01EW38Z38E2TA8CDEJNJ130QV9 01EW3FTMKQYF12V439SBT6B17P >> > >> 01EW3PPHPZ6ZZM2JGKF69F96RN]" duration=827.624569ms >> > >> level=info ts=2021-01-15T21:00:14.381Z caller=db.go:1187 >> component=tsdb >> > >> msg="Deleting obsolete block" block=01EW38Z38E2TA8CDEJNJ130QV9 >> > >> level=info ts=2021-01-15T21:00:14.547Z caller=db.go:1187 >> component=tsdb >> > >> msg="Deleting obsolete block" block=01EW3FTMKQYF12V439SBT6B17P >> > >> level=info ts=2021-01-15T21:00:14.705Z caller=db.go:1187 >> component=tsdb >> > >> msg="Deleting obsolete block" block=01EW3PPHPZ6ZZM2JGKF69F96RN >> > >> level=error ts=2021-01-15T21:03:15.152Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T21:20:19.710Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T21:36:24.045Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T21:38:26.227Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T21:46:29.339Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T21:47:31.378Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T22:00:37.302Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T22:02:39.487Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T22:17:43.600Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T22:26:47.082Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T22:35:50.353Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T22:36:52.383Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T23:00:06.094Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: write >> new >> > >> tombstones file: rename >> > >> >> /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones.tmp >> > >> /prometheus/01EW44DYM6ZW84JN85YDMGWQAJ.tmp-for-creation/tombstones: >> > >> interrupted system call" >> > >> level=error ts=2021-01-15T23:00:07.633Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: write >> merged >> > >> meta: rename >> > >> >> /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json.tmp >> > >> /prometheus/01EW44E09PCPM0T33BGVG6SE7S.tmp-for-creation/meta.json: >> > >> interrupted system call" >> > >> level=error ts=2021-01-15T23:00:13.695Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: >> rename block >> > >> dir: rename /prometheus/01EW44E60NC3EGCETC25GBW492.tmp-for-creation >> > >> /prometheus/01EW44E60NC3EGCETC25GBW492: interrupted system call" >> > >> level=error ts=2021-01-15T23:00:18.059Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: >> populate >> > >> block: write chunks: preallocate: seek >> > >> >> /prometheus/01EW44EAMZKJ5JJ2E42WCM8V0Z.tmp-for-creation/chunks/000001.tmp: >> > >> interrupted system call" >> > >> level=error ts=2021-01-15T23:00:26.689Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: write >> merged >> > >> meta: rename >> > >> >> /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json.tmp >> > >> /prometheus/01EW44EJTBB0G6JNM4JBTZQMJ7.tmp-for-creation/meta.json: >> > >> interrupted system call" >> > >> level=error ts=2021-01-15T23:00:43.395Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: write >> new >> > >> tombstones file: rename >> > >> >> /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones.tmp >> > >> /prometheus/01EW44F321S34ESD4MCNJNW0H7.tmp-for-creation/tombstones: >> > >> interrupted system call" >> > >> level=info ts=2021-01-15T23:01:16.055Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610740800000 >> maxt=1610748000000 >> > >> ulid=01EW44G303K0DNBB2EDM8DX69E duration=659.365701ms >> > >> level=info ts=2021-01-15T23:01:16.353Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=23.148498ms >> > >> level=info ts=2021-01-15T23:01:16.514Z caller=checkpoint.go:95 >> > >> component=tsdb msg="Creating checkpoint" from_segment=4 to_segment=5 >> > >> mint=1610748000000 >> > >> level=info ts=2021-01-15T23:01:17.004Z caller=head.go:908 >> component=tsdb >> > >> msg="WAL checkpoint complete" first=4 last=5 duration=512.755478ms >> > >> level=error ts=2021-01-15T23:22:20.652Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T23:25:23.109Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T23:40:27.567Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-15T23:50:31.230Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T00:03:35.502Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T00:20:40.429Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T00:21:42.458Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T00:23:46.659Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T00:25:48.862Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T00:37:52.910Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T00:57:58.400Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T01:00:00.602Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=info ts=2021-01-16T01:00:06.068Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610748000000 >> maxt=1610755200000 >> > >> ulid=01EW4B9NW6T794HCZATQJZJ4SZ duration=686.024281ms >> > >> level=info ts=2021-01-16T01:00:06.429Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=22.475094ms >> > >> level=error ts=2021-01-16T01:00:06.563Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: head memory truncate: >> truncate >> > >> chunks.HeadReadWriter: remove /prometheus/chunks_head/000007: >> interrupted >> > >> system call" >> > >> level=error ts=2021-01-16T01:08:12.075Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T01:15:15.300Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T01:20:18.133Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T01:22:20.357Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T01:31:24.051Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T01:50:29.872Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T01:52:32.155Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T02:00:35.607Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T02:40:45.873Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T02:54:50.472Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="plan compaction: readdirent: no such >> file or >> > >> directory" >> > >> level=error ts=2021-01-16T03:00:06.014Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: write >> merged >> > >> meta: rename >> > >> >> /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json.tmp >> > >> /prometheus/01EW4J5D3QHBTDS1VMS0VM60VY.tmp-for-creation/meta.json: >> > >> interrupted system call" >> > >> level=error ts=2021-01-16T03:00:16.046Z caller=compact.go:541 >> > >> component=tsdb msg="removed tmp folder after failed compaction" >> > >> err="unlinkat >> /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation: >> > >> directory not empty" >> > >> level=error ts=2021-01-16T03:00:16.046Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: sync >> > >> /prometheus/01EW4J5Q09F3YJ4A1Z1SWDTFCA.tmp-for-creation/index: >> interrupted >> > >> system call" >> > >> level=error ts=2021-01-16T03:00:20.488Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: open >> chunk >> > >> writer: mkdir >> > >> /prometheus/01EW4J5VTSRT3KXABDFXETHK99.tmp-for-creation/chunks: >> interrupted >> > >> system call" >> > >> level=error ts=2021-01-16T03:00:25.202Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact head: persist head block: write >> new >> > >> tombstones file: rename >> > >> >> /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones.tmp >> > >> /prometheus/01EW4J5ZS8S3EVJT4KBS6WCJF3.tmp-for-creation/tombstones: >> > >> interrupted system call" >> > >> level=info ts=2021-01-16T03:00:33.853Z caller=compact.go:502 >> > >> component=tsdb msg="write block" mint=1610755200000 >> maxt=1610762400000 >> > >> ulid=01EW4J689JDY2GWPCPDNHD0DNJ duration=650.542029ms >> > >> level=info ts=2021-01-16T03:00:34.211Z caller=head.go:824 >> component=tsdb >> > >> msg="Head GC completed" duration=22.278794ms >> > >> level=error ts=2021-01-16T03:00:35.654Z caller=db.go:746 >> component=tsdb >> > >> msg="compaction failed" err="compact >> > >> [/prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR >> > >> /prometheus/01EW44G303K0DNBB2EDM8DX69E >> > >> /prometheus/01EW4B9NW6T794HCZATQJZJ4SZ]: 2 errors: rename block dir: >> rename >> > >> /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK.tmp-for-creation >> > >> /prometheus/01EW4J69MNDMCRR7RWT3Q1TMGK: interrupted system call; >> setting >> > >> compaction failed for block: /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR: >> rename >> > >> /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json.tmp >> > >> /prometheus/01EW3XJDQ6FXT5YDQNSWB6W6VR/meta.json: interrupted system >> call" >> > >> >> > >> >> > >> =============================== >> > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:360 >> msg="Starting >> > >> Prometheus" version="(version=2.23.0, branch=HEAD, >> > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" >> > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:365 >> > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, >> > >> date=20201126-10:56:17)" >> > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:366 >> > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec >> 10 >> > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" >> > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:367 >> > >> fd_limits="(soft=1048576, hard=1048576)" >> > >> level=info ts=2021-01-19T05:06:08.707Z caller=main.go:368 >> > >> vm_limits="(soft=unlimited, hard=unlimited)" >> > >> level=error ts=2021-01-19T05:06:08.726Z caller=query_logger.go:71 >> > >> component=activeQueryTracker msg="Failed to read query log file" >> err=EOF >> > >> unexpected fault address 0x7f3c06738000 >> > >> fatal error: fault >> > >> [signal SIGBUS: bus error code=0x2 addr=0x7f3c06738000 pc=0x470ea8] >> > >> >> > >> goroutine 1 [running]: >> > >> runtime.throw(0x29f4a25, 0x5) >> > >> /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00092b248 >> > >> sp=0xc00092b218 pc=0x4378b2 >> > >> runtime.sigpanic() >> > >> /usr/local/go/src/runtime/signal_unix.go:739 +0x485 fp=0xc00092b278 >> > >> sp=0xc00092b248 pc=0x44e085 >> > >> runtime.memmove(0x7f3c06738000, 0x29f1e48, 0x1) >> > >> /usr/local/go/src/runtime/memmove_amd64.s:148 +0x108 fp=0xc00092b280 >> > >> sp=0xc00092b278 pc=0x470ea8 >> > >> >> > >> >> github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff842a774d, >> >> >> > >> 0xb, 0x14, 0x30d9a40, 0xc00077ede0, 0x30d9a40) >> > >> /app/promql/query_logger.go:120 +0x3a5 fp=0xc00092b3d8 >> sp=0xc00092b280 >> > >> pc=0x20fac05 >> > >> main.main() >> > >> /app/cmd/prometheus/main.go:396 +0x53cc fp=0xc00092bf88 >> sp=0xc00092b3d8 >> > >> pc=0x21f9a0c >> > >> runtime.main() >> > >> /usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc00092bfe0 >> > >> sp=0xc00092bf88 pc=0x43a0a9 >> > >> runtime.goexit() >> > >> /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00092bfe8 >> > >> sp=0xc00092bfe0 pc=0x46ffa1 >> > >> >> > >> goroutine 9 [select]: >> > >> go.opencensus.io/stats/view.(*worker).start(0xc00030c280) >> > >> /app/vendor/go.opencensus.io/stats/view/worker.go:276 +0x105 >> > >> created by go.opencensus.io/stats/view.init.0 >> > >> /app/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68 >> > >> >> > >> goroutine 85 [select]: >> > >> >> github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc000276780) >> > >> /app/pkg/logging/dedupe.go:75 +0x1e5 >> > >> created by github.com/prometheus/prometheus/pkg/logging.Dedupe >> > >> /app/pkg/logging/dedupe.go:61 +0xcf >> > >> >> > >> goroutine 92 [chan receive]: >> > >> >> > >> >> github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc00018ad20) >> >> > >> /app/storage/remote/write.go:96 +0xb6 >> > >> created by >> > >> github.com/prometheus/prometheus/storage/remote.NewWriteStorage >> > >> /app/storage/remote/write.go:89 +0x305 >> > >> >> > >> >> ============================================================================ >> >> > >> level=info ts=2021-01-19T05:42:16.710Z caller=main.go:360 >> msg="Starting >> > >> Prometheus" version="(version=2.23.0, branch=HEAD, >> > >> revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)" >> > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:365 >> > >> build_context="(go=go1.15.5, user=root@37609b3a0a21, >> > >> date=20201126-10:56:17)" >> > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:366 >> > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec >> 10 >> > >> 09:13:52 UTC 2020 x86_64 prometheus-64bbc6d46c-j5kjd (none))" >> > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:367 >> > >> fd_limits="(soft=1048576, hard=1048576)" >> > >> level=info ts=2021-01-19T05:42:16.711Z caller=main.go:368 >> > >> vm_limits="(soft=unlimited, hard=unlimited)" >> > >> level=info ts=2021-01-19T05:42:16.796Z caller=web.go:528 >> component=web >> > >> msg="Start listening for connections" address=0.0.0.0:9090 >> > >> level=info ts=2021-01-19T05:42:16.796Z caller=main.go:722 >> msg="Starting >> > >> TSDB ..." >> > >> level=info ts=2021-01-19T05:42:16.844Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610695556030 maxt=1610697600000 >> > >> ulid=01EW2S9SMSDCR35G93DT27G6NR >> > >> level=info ts=2021-01-19T05:42:16.875Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610697600000 maxt=1610704800000 >> > >> ulid=01EW2V7K5TDBRGNWYZXMMN50C7 >> > >> level=info ts=2021-01-19T05:42:16.901Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610704800000 maxt=1610712000000 >> > >> ulid=01EW323KD7FYEMJ0V03RX7TDR7 >> > >> level=info ts=2021-01-19T05:42:16.925Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610733600000 maxt=1610740800000 >> > >> ulid=01EW3XJDQ6FXT5YDQNSWB6W6VR >> > >> level=info ts=2021-01-19T05:42:16.951Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610712000000 maxt=1610733600000 >> > >> ulid=01EW3XJEZR86TBBDJFVTBNZ8VF >> > >> level=info ts=2021-01-19T05:42:16.976Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610740800000 maxt=1610748000000 >> > >> ulid=01EW44G303K0DNBB2EDM8DX69E >> > >> level=info ts=2021-01-19T05:42:17.001Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610748000000 maxt=1610755200000 >> > >> ulid=01EW4B9NW6T794HCZATQJZJ4SZ >> > >> level=info ts=2021-01-19T05:42:17.049Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610755200000 maxt=1610762400000 >> > >> ulid=01EW4J689JDY2GWPCPDNHD0DNJ >> > >> level=info ts=2021-01-19T05:42:17.074Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610762400000 maxt=1610769600000 >> > >> ulid=01EW4S14CHH73BZHP8152M3594 >> > >> level=info ts=2021-01-19T05:42:17.099Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610769600000 maxt=1610776800000 >> > >> ulid=01EW4ZXAASHGVGEY6QZ4C6R47M >> > >> level=info ts=2021-01-19T05:42:17.125Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610798400000 maxt=1610805600000 >> > >> ulid=01EW5VBMBQBYV98Q278AHHNK70 >> > >> level=info ts=2021-01-19T05:42:17.155Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610776800000 maxt=1610798400000 >> > >> ulid=01EW5VBP3B7KSWNETV0AEKFSWY >> > >> level=info ts=2021-01-19T05:42:17.184Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610805600000 maxt=1610812800000 >> > >> ulid=01EW627M0DHYWTCC0N237SC62D >> > >> level=info ts=2021-01-19T05:42:17.234Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610812800000 maxt=1610820000000 >> > >> ulid=01EW6939M4CNXWMHGT34SD5N6R >> > >> level=info ts=2021-01-19T05:42:17.259Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610820000000 maxt=1610841600000 >> > >> ulid=01EW74JPT1T99G30CWTAE8YTVC >> > >> level=info ts=2021-01-19T05:42:17.285Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610863200000 maxt=1610870400000 >> > >> ulid=01EW7S59W18PRNC14KHMY2NQDF >> > >> level=info ts=2021-01-19T05:42:17.311Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610841600000 maxt=1610863200000 >> > >> ulid=01EW7S5CEZAKV5FHJD1BYN7VFS >> > >> level=info ts=2021-01-19T05:42:17.337Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610870400000 maxt=1610877600000 >> > >> ulid=01EW8012X70MJZYVQTEKERJSF0 >> > >> level=info ts=2021-01-19T05:42:17.393Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610877600000 maxt=1610884800000 >> > >> ulid=01EW86X36BN7KDN3YQY4KH1X7Z >> > >> level=info ts=2021-01-19T05:42:17.418Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610884800000 maxt=1610892000000 >> > >> ulid=01EW8DRFQEXQP7TCA6W9ZCWJJC >> > >> level=info ts=2021-01-19T05:42:17.443Z caller=repair.go:57 >> component=tsdb >> > >> msg="Found healthy block" mint=1610863200000 maxt=1610884800000 >> > >> ulid=01EW8DRHV7JX5CV9YKDFYBQQ99 >> > >> level=info ts=2021-01-19T05:42:17.606Z caller=main.go:595 >> msg="Stopping >> > >> scrape discovery manager..." >> > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:609 >> msg="Stopping >> > >> notify discovery manager..." >> > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:631 >> msg="Stopping >> > >> scrape manager..." >> > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:605 >> msg="Notify >> > >> discovery manager stopped" >> > >> level=info ts=2021-01-19T05:42:17.607Z caller=main.go:591 >> msg="Scrape >> > >> discovery manager stopped" >> > >> level=info ts=2021-01-19T05:42:17.607Z caller=manager.go:924 >> > >> component="rule manager" msg="Stopping rule manager..." >> > >> level=info ts=2021-01-19T05:42:17.608Z caller=manager.go:934 >> > >> component="rule manager" msg="Rule manager stopped" >> > >> level=info ts=2021-01-19T05:42:17.608Z caller=notifier.go:601 >> > >> component=notifier msg="Stopping notification manager..." >> > >> level=info ts=2021-01-19T05:42:17.608Z caller=main.go:799 >> msg="Notifier >> > >> manager stopped" >> > >> level=info ts=2021-01-19T05:42:17.608Z caller=main.go:625 >> msg="Scrape >> > >> manager stopped" >> > >> level=error ts=2021-01-19T05:42:17.608Z caller=main.go:808 >> err="opening >> > >> storage failed: found unsequential head chunk files >> > >> /prometheus/chunks_head/000007 (index: 7) and >> > >> /prometheus/chunks_head/000027 (index: 27)" >> > >> ======================================= >> > >> level=info ts=2021-01-19T10:55:57.643Z caller=main.go:337 >> msg="Starting >> > >> Prometheus" version="(version=2.19.1, branch=HEAD, >> > >> revision=eba3fdcbf0d378b66600281903e3aab515732b39)" >> > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:338 >> > >> build_context="(go=go1.14.4, user=root@62700b3d0ef9, >> > >> date=20200618-16:35:26)" >> > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:339 >> > >> host_details="(Linux 5.4.0-1034-azure #35~18.04.1-Ubuntu SMP Thu Dec >> 10 >> > >> 09:13:52 UTC 2020 x86_64 prometheus-84fb48b4d9-ddq9m (none))" >> > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:340 >> > >> fd_limits="(soft=1048576, hard=1048576)" >> > >> level=info ts=2021-01-19T10:55:57.644Z caller=main.go:341 >> > >> vm_limits="(soft=unlimited, hard=unlimited)" >> > >> level=error ts=2021-01-19T10:55:57.704Z caller=query_logger.go:93 >> > >> component=activeQueryTracker msg="Error setting filesize." >> filesize=20001 >> > >> err="truncate /prometheus/queries.active: interrupted system call" >> > >> panic: Unable to create mmap-ed active query log >> > >> >> > >> goroutine 1 [running]: >> > >> >> > >> >> github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff2f2aa74d, >> >> >> > >> 0xb, 0x14, 0x2f3b7e0, 0xc00074d800, 0x2f3b7e0) >> > >> /app/promql/query_logger.go:117 +0x4cd >> > >> main.main() >> > >> /app/cmd/prometheus/main.go:368 +0x4ed2 >> > >> >> > >> >> > >> Appreciate any help >> > >> >> > >> Thanks. >> > >> >> > > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/dd2528cb-522d-44ac-900a-af2a53ad431bn%40googlegroups.com. >> >> >> >> >> -- >> Julien Pivotto >> @roidelapluie >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6f3d029d-2b4f-4dc3-8b3f-c6369820ad8en%40googlegroups.com.
