"Out of order sample from remote write". You have remote write clients that are sending out of order/overlapping data.
https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus/ This is not likely to cause a crash IIRC. Have you checked to see if you have allocated enough memory to your Prometheus? What does Kubernetes say is the crash reason? In all probability, this is simply an OOM issue. Give your instance more memory. On Tue, May 23, 2023 at 10:55 AM Nick Carlton <nick.carlton...@gmail.com> wrote: > Hello, > > I have a Prometheus deployment in Azure using the kube-prometheus-stack > helm chart which receives lots of remote writes. This runs fine normally > but every couple of months or so, will randomly crash and start this crash > loop cycle. It does usually self-resolve but takes Prom out for a couple of > days. > > We use a PVC with Prometheus with a size of 200GB and with the access mode > "ReadWriteOnce". > > Basically what I am seeing happen is, Prometheus Starts, complains about > some storage chunks and does a WAL replay. Then it does start, however, we > get thousands of messages about remote write "out of bounds" and then it > crashes and loops again. Its will continue like this even if the pods are > deleted and re-created. > > See Logs: > > ts=2023-05-23T08:12:03.968Z caller=main.go:556 level=info msg="Starting > Prometheus Server" mode=server version="(version=2.42.0, branch=HEAD, > revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" > ts=2023-05-23T08:12:03.968Z caller=main.go:561 level=info > build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, > date=20230201-07:53:32)" ts=2023-05-23T08:12:03.968Z caller=main.go:562 > level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu > May 19 18:52:16 UTC 2022 x86_64 > prometheus-prometheus-kube-prometheus-prometheus-0 (none))" > ts=2023-05-23T08:12:03.968Z caller=main.go:563 level=info > fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:12:03.968Z > caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" > ts=2023-05-23T08:12:03.970Z caller=web.go:561 level=info component=web > msg="Start listening for connections" address=0.0.0.0:9090 > ts=2023-05-23T08:12:03.970Z caller=main.go:993 level=info msg="Starting > TSDB ..." ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info > component=tsdb msg="Found healthy block" mint=1683892800000 > maxt=1683957600000 ulid=01H0A5J8MWQ152H85DYY6F32PK > ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1683957600000 maxt=1684022400000 > ulid=01H0C38Y02M29M82TXK2V7NN7G ts=2023-05-23T08:12:03.970Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684022400000 maxt=1684087200000 ulid=01H0E12DTC6PWH4J6R7B1H36XW > ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684087200003 maxt=1684152000000 > ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684152000003 maxt=1684216800000 ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684216800000 maxt=1684281600000 > ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684281600003 maxt=1684346400000 ulid=01H0NZ78AACK9BPD034VWH7SJS > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684346400000 maxt=1684411200000 > ulid=01H0QP2AA05C33ZVAAVDBT2E43 ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684411200004 maxt=1684476000000 ulid=01H0STTM76GGEKKYAMWV1KNTPT > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684476000004 maxt=1684540800000 > ulid=01H0VRMBDAC5D5M6TEVHRP80PS ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684540800043 maxt=1684605600000 ulid=01H0XPDYZ3D2X8XJ97B97M4BDH > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684605600043 maxt=1684670400000 > ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684735200044 maxt=1684742400000 ulid=01H11AZP33QHEGWR2631MYMY3M > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684742400044 maxt=1684749600000 > ulid=01H11HVD455FTTADM149WAHAH0 ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684670400032 maxt=1684735200000 ulid=01H11J11TMN4XWACE1Q9AJ1KXC > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684749600043 maxt=1684756800000 > ulid=01H11RQ4AHY8PVQRCM58CKTVZF ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684756800043 maxt=1684764000000 ulid=01H11ZJVV4E276NWR3370HFN00 > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684764000000 maxt=1684771200000 > ulid=01H12712JWFQMYW8FR8YXGQN8Y ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684771200000 maxt=1684778400000 ulid=01H12DAKVHHHGEY3TKY0FPQSVZ > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684778400013 maxt=1684785600000 > ulid=01H12M7CW13EEQRVRN6J4ZD7AK ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684785600070 maxt=1684792800000 ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684792800033 maxt=1684800000000 > ulid=01H131XPKE4VB6RMRQF3S67M0Y ts=2023-05-23T08:12:03.971Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684800000010 maxt=1684807200000 ulid=01H138S7B8Y8Q14YKJZ5SRXTAX > ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684807200000 maxt=1684814400000 > ulid=01H13FXP6EB7T1KSRR8P5GMFP3 ts=2023-05-23T08:12:03.971Z > caller=tls_config.go:232 level=info component=web msg="Listening on" > address=[::]:9090 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 > level=info component=tsdb msg="Found healthy block" mint=1684814400043 > maxt=1684821600000 ulid=01H13PQXNNJNH4GETATBWHPGYJ > ts=2023-05-23T08:12:03.971Z caller=tls_config.go:271 level=info > component=web msg="TLS is disabled." http2=false address=[::]:9090 > ts=2023-05-23T08:12:04.257Z caller=head.go:564 level=info component=tsdb > msg="Replaying on-disk memory mappable chunks if any" > ts=2023-05-23T08:12:04.318Z caller=head.go:592 level=error component=tsdb > msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of > sequence m-mapped chunk for series ref 19360078, last chunk: > [1684824583256, 1684825285256], new: [1684822459256, 1684823161256]" > ts=2023-05-23T08:12:04.318Z caller=head.go:831 level=info component=tsdb > msg="Deleting mmapped chunk files" ts=2023-05-23T08:12:04.319Z > caller=head.go:837 level=info component=tsdb msg="Deleting mmapped chunk > files" ts=2023-05-23T08:12:04.319Z caller=head.go:840 level=info > component=tsdb msg="Deletion of corrupted mmap chunk files failed, > discarding chunk files completely" err="cannot handle error: iterate on > on-disk chunks: out of sequence m-mapped chunk for series ref 19360078, > last chunk: [1684824583256, 1684825285256], new: [1684822459256, > 1684823161256]" ts=2023-05-23T08:12:04.325Z caller=head.go:608 level=info > component=tsdb msg="On-disk memory mappable chunks replay completed" > duration=67.431625ms ts=2023-05-23T08:12:04.325Z caller=head.go:614 > level=info component=tsdb msg="Replaying WAL, this may take a while" > ts=2023-05-23T08:12:14.136Z caller=head.go:650 level=info component=tsdb > msg="WAL checkpoint loaded" ts=2023-05-23T08:12:14.145Z caller=head.go:685 > level=info component=tsdb msg="WAL segment loaded" segment=17588 > maxSegment=17630 ts=2023-05-23T08:12:14.197Z caller=head.go:685 level=info > component=tsdb msg="WAL segment loaded" segment=17589 maxSegment=17630 > ts=2023-05-23T08:12:14.216Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17590 maxSegment=17630 > ts=2023-05-23T08:12:14.250Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17591 maxSegment=17630 > ts=2023-05-23T08:12:14.293Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17592 maxSegment=17630 > ts=2023-05-23T08:12:15.026Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17593 maxSegment=17630 > ts=2023-05-23T08:12:15.063Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17594 maxSegment=17630 > ts=2023-05-23T08:12:15.188Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17595 maxSegment=17630 > ts=2023-05-23T08:12:15.287Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17596 maxSegment=17630 > ts=2023-05-23T08:12:15.378Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17597 maxSegment=17630 > ts=2023-05-23T08:12:15.457Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17598 maxSegment=17630 > ts=2023-05-23T08:12:15.543Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17599 maxSegment=17630 > ts=2023-05-23T08:12:15.619Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17600 maxSegment=17630 > ts=2023-05-23T08:12:15.745Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17601 maxSegment=17630 > ts=2023-05-23T08:12:15.890Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17602 maxSegment=17630 > ts=2023-05-23T08:12:15.977Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17603 maxSegment=17630 > ts=2023-05-23T08:12:16.057Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17604 maxSegment=17630 > ts=2023-05-23T08:12:16.156Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17605 maxSegment=17630 > ts=2023-05-23T08:12:16.157Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17606 maxSegment=17630 > ts=2023-05-23T08:12:17.091Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17607 maxSegment=17630 > ts=2023-05-23T08:12:18.271Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17608 maxSegment=17630 > ts=2023-05-23T08:12:18.449Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17609 maxSegment=17630 > ts=2023-05-23T08:12:18.787Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17610 maxSegment=17630 > ts=2023-05-23T08:12:19.671Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17611 maxSegment=17630 > ts=2023-05-23T08:12:19.894Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17612 maxSegment=17630 > ts=2023-05-23T08:12:20.178Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17613 maxSegment=17630 > ts=2023-05-23T08:12:20.344Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17614 maxSegment=17630 > ts=2023-05-23T08:12:22.299Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17616 maxSegment=17630 > ts=2023-05-23T08:12:23.180Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17617 maxSegment=17630 > ts=2023-05-23T08:12:23.343Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17618 maxSegment=17630 > ts=2023-05-23T08:12:23.599Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17619 maxSegment=17630 > ts=2023-05-23T08:12:23.764Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17620 maxSegment=17630 > ts=2023-05-23T08:12:24.128Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17621 maxSegment=17630 > ts=2023-05-23T08:12:24.317Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17622 maxSegment=17630 > ts=2023-05-23T08:12:24.463Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17623 maxSegment=17630 > ts=2023-05-23T08:12:24.809Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17624 maxSegment=17630 > ts=2023-05-23T08:12:25.088Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17625 maxSegment=17630 > ts=2023-05-23T08:12:25.360Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17626 maxSegment=17630 > ts=2023-05-23T08:12:25.642Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17627 maxSegment=17630 > ts=2023-05-23T08:12:25.664Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17628 maxSegment=17630 > ts=2023-05-23T08:12:25.884Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17629 maxSegment=17630 > ts=2023-05-23T08:12:25.884Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17630 maxSegment=17630 > ts=2023-05-23T08:12:25.884Z caller=head.go:722 level=info component=tsdb > msg="WAL replay completed" checkpoint_replay_duration=9.811449959s > wal_replay_duration=11.747824132s wbl_replay_duration=215ns > total_replay_duration=21.626738156s ts=2023-05-23T08:12:28.177Z > caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC > ts=2023-05-23T08:12:28.177Z caller=main.go:1017 level=info msg="TSDB > started" ts=2023-05-23T08:12:28.177Z caller=main.go:1197 level=info > msg="Loading configuration file" > filename=/etc/prometheus/config_out/prometheus.env.yaml > ts=2023-05-23T08:12:28.181Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-operator/0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info > component="discovery manager notify" discovery=kubernetes config=config-0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:12:28.221Z caller=main.go:1234 level=info msg="Completed > loading of configuration file" > filename=/etc/prometheus/config_out/prometheus.env.yaml > totalDuration=43.976379ms db_storage=1.549µs remote_storage=1.501µs > web_handler=594ns query_engine=1.035µs scrape=207.016µs scrape_sd=565.387µs > notify=13.403µs notify_sd=116.569µs rules=39.2593ms tracing=5.19µs > ts=2023-05-23T08:12:28.221Z caller=main.go:978 level=info msg="Server is > ready to receive web requests." ts=2023-05-23T08:12:28.221Z > caller=manager.go:974 level=info component="rule manager" msg="Starting > rule manager..." ts=2023-05-23T08:13:17.453Z caller=write_handler.go:109 > level=error component=web msg="Out of order sample from remote write" > err="out of bounds" series ... timestamp=1684825890472 ... > ts=2023-05-23T08:13:17.547Z caller=write_handler.go:109 level=error > component=web msg="Out of order sample from remote write" err="out of > bounds" series ... timestamp=1684808929971 ts=2023-05-23T08:13:29.090Z > caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server > version="(version=2.42.0, branch=HEAD, > revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" > ts=2023-05-23T08:13:29.090Z caller=main.go:561 level=info > build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, > date=20230201-07:53:32)" ts=2023-05-23T08:13:29.090Z caller=main.go:562 > level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu > May 19 18:52:16 UTC 2022 x86_64 > prometheus-prometheus-kube-prometheus-prometheus-0 (none))" > ts=2023-05-23T08:13:29.090Z caller=main.go:563 level=info > fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:13:29.090Z > caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" > ts=2023-05-23T08:13:29.090Z caller=query_logger.go:80 level=info > component=activeQueryTracker msg="These queries didn't finish in > prometheus' last run:" queries=[829607] ts=2023-05-23T08:13:29.091Z > caller=web.go:561 level=info component=web msg="Start listening for > connections" address=0.0.0.0:9090 ts=2023-05-23T08:13:29.092Z > caller=main.go:993 level=info msg="Starting TSDB ..." > ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1683892800000 maxt=1683957600000 > ulid=01H0A5J8MWQ152H85DYY6F32PK ts=2023-05-23T08:13:29.092Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1683957600000 maxt=1684022400000 ulid=01H0C38Y02M29M82TXK2V7NN7G > ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684022400000 maxt=1684087200000 > ulid=01H0E12DTC6PWH4J6R7B1H36XW ts=2023-05-23T08:13:29.092Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684087200003 maxt=1684152000000 ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ > ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684152000003 maxt=1684216800000 > ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684216800000 maxt=1684281600000 ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684281600003 maxt=1684346400000 > ulid=01H0NZ78AACK9BPD034VWH7SJS ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684346400000 maxt=1684411200000 ulid=01H0QP2AA05C33ZVAAVDBT2E43 > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684411200004 maxt=1684476000000 > ulid=01H0STTM76GGEKKYAMWV1KNTPT ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684476000004 maxt=1684540800000 ulid=01H0VRMBDAC5D5M6TEVHRP80PS > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684540800043 maxt=1684605600000 > ulid=01H0XPDYZ3D2X8XJ97B97M4BDH ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684605600043 maxt=1684670400000 ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 > ts=2023-05-23T08:13:29.093Z caller=tls_config.go:232 level=info > component=web msg="Listening on" address=[::]:9090 > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684735200044 maxt=1684742400000 > ulid=01H11AZP33QHEGWR2631MYMY3M ts=2023-05-23T08:13:29.093Z > caller=tls_config.go:271 level=info component=web msg="TLS is disabled." > http2=false address=[::]:9090 ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684742400044 maxt=1684749600000 ulid=01H11HVD455FTTADM149WAHAH0 > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684670400032 maxt=1684735200000 > ulid=01H11J11TMN4XWACE1Q9AJ1KXC ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684749600043 maxt=1684756800000 ulid=01H11RQ4AHY8PVQRCM58CKTVZF > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684756800043 maxt=1684764000000 > ulid=01H11ZJVV4E276NWR3370HFN00 ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684764000000 maxt=1684771200000 ulid=01H12712JWFQMYW8FR8YXGQN8Y > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684771200000 maxt=1684778400000 > ulid=01H12DAKVHHHGEY3TKY0FPQSVZ ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684778400013 maxt=1684785600000 ulid=01H12M7CW13EEQRVRN6J4ZD7AK > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684785600070 maxt=1684792800000 > ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684792800033 maxt=1684800000000 ulid=01H131XPKE4VB6RMRQF3S67M0Y > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684800000010 maxt=1684807200000 > ulid=01H138S7B8Y8Q14YKJZ5SRXTAX ts=2023-05-23T08:13:29.093Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684807200000 maxt=1684814400000 ulid=01H13FXP6EB7T1KSRR8P5GMFP3 > ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684814400043 maxt=1684821600000 > ulid=01H13PQXNNJNH4GETATBWHPGYJ ts=2023-05-23T08:13:29.365Z > caller=head.go:564 level=info component=tsdb msg="Replaying on-disk memory > mappable chunks if any" ts=2023-05-23T08:13:29.426Z caller=head.go:608 > level=info component=tsdb msg="On-disk memory mappable chunks replay > completed" duration=60.526867ms ts=2023-05-23T08:13:29.426Z > caller=head.go:614 level=info component=tsdb msg="Replaying WAL, this may > take a while" ts=2023-05-23T08:13:38.994Z caller=head.go:650 level=info > component=tsdb msg="WAL checkpoint loaded" ts=2023-05-23T08:13:39.003Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17588 maxSegment=17631 ts=2023-05-23T08:13:39.038Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17589 maxSegment=17631 ts=2023-05-23T08:13:39.055Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17590 maxSegment=17631 ts=2023-05-23T08:13:39.081Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17591 maxSegment=17631 ts=2023-05-23T08:13:39.120Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17592 maxSegment=17631 ts=2023-05-23T08:13:40.778Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17593 maxSegment=17631 ts=2023-05-23T08:13:40.811Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17594 maxSegment=17631 ts=2023-05-23T08:13:40.928Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17595 maxSegment=17631 ts=2023-05-23T08:13:41.015Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17596 maxSegment=17631 ts=2023-05-23T08:13:41.091Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17597 maxSegment=17631 ts=2023-05-23T08:13:41.158Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17598 maxSegment=17631 ts=2023-05-23T08:13:41.234Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17599 maxSegment=17631 ts=2023-05-23T08:13:41.300Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17600 maxSegment=17631 ts=2023-05-23T08:13:41.406Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17601 maxSegment=17631 ts=2023-05-23T08:13:41.543Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17602 maxSegment=17631 ts=2023-05-23T08:13:41.619Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17603 maxSegment=17631 ts=2023-05-23T08:13:41.689Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17604 maxSegment=17631 ts=2023-05-23T08:13:41.770Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17605 maxSegment=17631 ts=2023-05-23T08:13:41.771Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17606 maxSegment=17631 ts=2023-05-23T08:13:42.359Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17607 maxSegment=17631 ts=2023-05-23T08:13:42.843Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17608 maxSegment=17631 ts=2023-05-23T08:13:43.023Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17609 maxSegment=17631 ts=2023-05-23T08:13:43.362Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17610 maxSegment=17631 ts=2023-05-23T08:13:44.268Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17611 maxSegment=17631 ts=2023-05-23T08:13:44.500Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17612 maxSegment=17631 ts=2023-05-23T08:13:44.792Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17613 maxSegment=17631 ts=2023-05-23T08:13:44.958Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17614 maxSegment=17631 ts=2023-05-23T08:13:45.175Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17615 maxSegment=17631 ts=2023-05-23T08:13:46.881Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17616 maxSegment=17631 ts=2023-05-23T08:13:48.160Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17617 maxSegment=17631 ts=2023-05-23T08:13:48.529Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17618 maxSegment=17631 ts=2023-05-23T08:13:49.071Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17619 maxSegment=17631 ts=2023-05-23T08:13:49.463Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17620 maxSegment=17631 ts=2023-05-23T08:13:49.864Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17621 maxSegment=17631 ts=2023-05-23T08:13:50.052Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17622 maxSegment=17631 ts=2023-05-23T08:13:50.181Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17623 maxSegment=17631 ts=2023-05-23T08:13:51.235Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17627 maxSegment=17631 ts=2023-05-23T08:13:51.258Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17628 maxSegment=17631 ts=2023-05-23T08:13:51.476Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17629 maxSegment=17631 ts=2023-05-23T08:13:51.707Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17630 maxSegment=17631 ts=2023-05-23T08:13:51.707Z > caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" > segment=17631 maxSegment=17631 ts=2023-05-23T08:13:51.707Z > caller=head.go:722 level=info component=tsdb msg="WAL replay completed" > checkpoint_replay_duration=9.568658965s wal_replay_duration=12.712461039s > wbl_replay_duration=240ns total_replay_duration=22.341708746s > ts=2023-05-23T08:13:54.095Z caller=main.go:1014 level=info > fs_type=EXT4_SUPER_MAGIC ts=2023-05-23T08:13:54.095Z caller=main.go:1017 > level=info msg="TSDB started" ts=2023-05-23T08:13:54.095Z > caller=main.go:1197 level=info msg="Loading configuration file" > filename=/etc/prometheus/config_out/prometheus.env.yaml > ts=2023-05-23T08:13:54.098Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/2 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-prometheus/0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info > component="discovery manager notify" discovery=kubernetes config=config-0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:13:54.136Z caller=main.go:1234 level=info msg="Completed > loading of configuration file" > filename=/etc/prometheus/config_out/prometheus.env.yaml > totalDuration=41.309545ms db_storage=946ns remote_storage=1.338µs > web_handler=633ns query_engine=1.422µs scrape=229.86µs scrape_sd=593.335µs > notify=15.321µs notify_sd=186.832µs rules=36.799825ms tracing=4.126µs > ts=2023-05-23T08:13:54.136Z caller=main.go:978 level=info msg="Server is > ready to receive web requests." ts=2023-05-23T08:13:54.136Z > caller=manager.go:974 level=info component="rule manager" msg="Starting > rule manager..." ts=2023-05-23T08:13:58.936Z caller=write_handler.go:109 > level=error component=web msg="Out of order sample from remote write" > err="out of bounds" series ... timestamp=1684808929971 ... > ts=2023-05-23T08:14:16.023Z caller=write_handler.go:109 level=error > component=web msg="Out of order sample from remote write" err="out of > bounds" series ... timestamp=1684825987289 ts=2023-05-23T08:14:34.749Z > caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server > version="(version=2.42.0, branch=HEAD, > revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" > ts=2023-05-23T08:14:34.749Z caller=main.go:561 level=info > build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, > date=20230201-07:53:32)" ts=2023-05-23T08:14:34.749Z caller=main.go:562 > level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu > May 19 18:52:16 UTC 2022 x86_64 > prometheus-prometheus-kube-prometheus-prometheus-0 (none))" > ts=2023-05-23T08:14:34.749Z caller=main.go:563 level=info > fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:14:34.749Z > caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" > ts=2023-05-23T08:14:34.750Z caller=web.go:561 level=info component=web > msg="Start listening for connections" address=0.0.0.0:9090 > ts=2023-05-23T08:14:34.752Z caller=main.go:993 level=info msg="Starting > TSDB ..." ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info > component=tsdb msg="Found healthy block" mint=1683892800000 > maxt=1683957600000 ulid=01H0A5J8MWQ152H85DYY6F32PK > ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1683957600000 maxt=1684022400000 > ulid=01H0C38Y02M29M82TXK2V7NN7G ts=2023-05-23T08:14:34.752Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684022400000 maxt=1684087200000 ulid=01H0E12DTC6PWH4J6R7B1H36XW > ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684087200003 maxt=1684152000000 > ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ ts=2023-05-23T08:14:34.752Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684152000003 maxt=1684216800000 ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 > ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684216800000 maxt=1684281600000 > ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 ts=2023-05-23T08:14:34.752Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684281600003 maxt=1684346400000 ulid=01H0NZ78AACK9BPD034VWH7SJS > ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684346400000 maxt=1684411200000 > ulid=01H0QP2AA05C33ZVAAVDBT2E43 ts=2023-05-23T08:14:34.752Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684411200004 maxt=1684476000000 ulid=01H0STTM76GGEKKYAMWV1KNTPT > ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684476000004 maxt=1684540800000 > ulid=01H0VRMBDAC5D5M6TEVHRP80PS ts=2023-05-23T08:14:34.753Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684540800043 maxt=1684605600000 ulid=01H0XPDYZ3D2X8XJ97B97M4BDH > ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684605600043 maxt=1684670400000 > ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 ts=2023-05-23T08:14:34.753Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684735200044 maxt=1684742400000 ulid=01H11AZP33QHEGWR2631MYMY3M > ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684742400044 maxt=1684749600000 > ulid=01H11HVD455FTTADM149WAHAH0 ts=2023-05-23T08:14:34.753Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684670400032 maxt=1684735200000 ulid=01H11J11TMN4XWACE1Q9AJ1KXC > ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684749600043 maxt=1684756800000 > ulid=01H11RQ4AHY8PVQRCM58CKTVZF ts=2023-05-23T08:14:34.753Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684756800043 maxt=1684764000000 ulid=01H11ZJVV4E276NWR3370HFN00 > ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684764000000 maxt=1684771200000 > ulid=01H12712JWFQMYW8FR8YXGQN8Y ts=2023-05-23T08:14:34.753Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684771200000 maxt=1684778400000 ulid=01H12DAKVHHHGEY3TKY0FPQSVZ > ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684778400013 maxt=1684785600000 > ulid=01H12M7CW13EEQRVRN6J4ZD7AK ts=2023-05-23T08:14:34.753Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684785600070 maxt=1684792800000 ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM > ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684792800033 maxt=1684800000000 > ulid=01H131XPKE4VB6RMRQF3S67M0Y ts=2023-05-23T08:14:34.753Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684800000010 maxt=1684807200000 ulid=01H138S7B8Y8Q14YKJZ5SRXTAX > ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb > msg="Found healthy block" mint=1684807200000 maxt=1684814400000 > ulid=01H13FXP6EB7T1KSRR8P5GMFP3 ts=2023-05-23T08:14:34.753Z > caller=repair.go:56 level=info component=tsdb msg="Found healthy block" > mint=1684814400043 maxt=1684821600000 ulid=01H13PQXNNJNH4GETATBWHPGYJ > ts=2023-05-23T08:14:34.753Z caller=tls_config.go:232 level=info > component=web msg="Listening on" address=[::]:9090 > ts=2023-05-23T08:14:34.753Z caller=tls_config.go:271 level=info > component=web msg="TLS is disabled." http2=false address=[::]:9090 > ts=2023-05-23T08:14:35.063Z caller=head.go:564 level=info component=tsdb > msg="Replaying on-disk memory mappable chunks if any" > ts=2023-05-23T08:14:35.125Z caller=head.go:592 level=error component=tsdb > msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of > sequence m-mapped chunk for series ref 19360078, last chunk: > [1684824583256, 1684825285256], new: [1684822459256, 1684823161256]" > ts=2023-05-23T08:14:35.125Z caller=head.go:831 level=info component=tsdb > msg="Deleting mmapped chunk files" ts=2023-05-23T08:14:35.129Z > caller=head.go:837 level=info component=tsdb msg="Deleting mmapped chunk > files" ts=2023-05-23T08:14:35.129Z caller=head.go:840 level=info > component=tsdb msg="Deletion of corrupted mmap chunk files failed, > discarding chunk files completely" err="cannot handle error: iterate on > on-disk chunks: out of sequence m-mapped chunk for series ref 19360078, > last chunk: [1684824583256, 1684825285256], new: [1684822459256, > 1684823161256]" ts=2023-05-23T08:14:35.135Z caller=head.go:608 level=info > component=tsdb msg="On-disk memory mappable chunks replay completed" > duration=71.923626ms ts=2023-05-23T08:14:35.135Z caller=head.go:614 > level=info component=tsdb msg="Replaying WAL, this may take a while" > ts=2023-05-23T08:14:44.651Z caller=head.go:650 level=info component=tsdb > msg="WAL checkpoint loaded" ts=2023-05-23T08:14:44.659Z caller=head.go:685 > level=info component=tsdb msg="WAL segment loaded" segment=17588 > maxSegment=17632 ts=2023-05-23T08:14:44.710Z caller=head.go:685 level=info > component=tsdb msg="WAL segment loaded" segment=17589 maxSegment=17632 > ts=2023-05-23T08:14:44.729Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17590 maxSegment=17632 > ts=2023-05-23T08:14:44.763Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17591 maxSegment=17632 > ts=2023-05-23T08:14:44.894Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17592 maxSegment=17632 > ts=2023-05-23T08:14:46.470Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17593 maxSegment=17632 > ts=2023-05-23T08:14:46.503Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17594 maxSegment=17632 > ts=2023-05-23T08:14:46.616Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17595 maxSegment=17632 > ts=2023-05-23T08:14:46.710Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17596 maxSegment=17632 > ts=2023-05-23T08:14:46.796Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17597 maxSegment=17632 > ts=2023-05-23T08:14:46.876Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17598 maxSegment=17632 > ts=2023-05-23T08:14:46.955Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17599 maxSegment=17632 > ts=2023-05-23T08:14:47.025Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17600 maxSegment=17632 > ts=2023-05-23T08:14:47.135Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17601 maxSegment=17632 > ts=2023-05-23T08:14:47.252Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17602 maxSegment=17632 > ts=2023-05-23T08:14:47.336Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17603 maxSegment=17632 > ts=2023-05-23T08:14:47.406Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17604 maxSegment=17632 > ts=2023-05-23T08:14:47.496Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17605 maxSegment=17632 > ts=2023-05-23T08:14:47.499Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17606 maxSegment=17632 > ts=2023-05-23T08:14:48.079Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17607 maxSegment=17632 > ts=2023-05-23T08:14:48.573Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17608 maxSegment=17632 > ts=2023-05-23T08:14:48.745Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17609 maxSegment=17632 > ts=2023-05-23T08:14:49.082Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17610 maxSegment=17632 > ts=2023-05-23T08:14:49.967Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17611 maxSegment=17632 > ts=2023-05-23T08:14:50.190Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17612 maxSegment=17632 > ts=2023-05-23T08:14:52.558Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17616 maxSegment=17632 > ts=2023-05-23T08:14:54.144Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17617 maxSegment=17632 > ts=2023-05-23T08:14:54.490Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17618 maxSegment=17632 > ts=2023-05-23T08:14:55.119Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17619 maxSegment=17632 > ts=2023-05-23T08:14:55.293Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17620 maxSegment=17632 > ts=2023-05-23T08:14:55.633Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17621 maxSegment=17632 > ts=2023-05-23T08:14:55.826Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17622 maxSegment=17632 > ts=2023-05-23T08:14:55.976Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17623 maxSegment=17632 > ts=2023-05-23T08:14:56.335Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17624 maxSegment=17632 > ts=2023-05-23T08:14:56.615Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17625 maxSegment=17632 > ts=2023-05-23T08:14:56.882Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17626 maxSegment=17632 > ts=2023-05-23T08:14:57.171Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17627 maxSegment=17632 > ts=2023-05-23T08:14:57.193Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17628 maxSegment=17632 > ts=2023-05-23T08:14:57.402Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17629 maxSegment=17632 > ts=2023-05-23T08:14:57.616Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17630 maxSegment=17632 > ts=2023-05-23T08:14:57.718Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17631 maxSegment=17632 > ts=2023-05-23T08:14:57.718Z caller=head.go:685 level=info component=tsdb > msg="WAL segment loaded" segment=17632 maxSegment=17632 > ts=2023-05-23T08:14:57.718Z caller=head.go:722 level=info component=tsdb > msg="WAL replay completed" checkpoint_replay_duration=9.515250193s > wal_replay_duration=13.067369985s wbl_replay_duration=205ns > total_replay_duration=22.654583208s ts=2023-05-23T08:15:00.133Z > caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC > ts=2023-05-23T08:15:00.133Z caller=main.go:1017 level=info msg="TSDB > started" ts=2023-05-23T08:15:00.134Z caller=main.go:1197 level=info > msg="Loading configuration file" > filename=/etc/prometheus/config_out/prometheus.env.yaml > ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-alertmanager/0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info > component="discovery manager scrape" discovery=kubernetes > config=serviceMonitor/monitoring/prometheus-kube-prometheus-kube-scheduler/0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info > component="discovery manager notify" discovery=kubernetes config=config-0 > msg="Using pod service account via in-cluster config" > ts=2023-05-23T08:15:00.176Z caller=main.go:1234 level=info msg="Completed > loading of configuration file" > filename=/etc/prometheus/config_out/prometheus.env.yaml > totalDuration=42.962628ms db_storage=1.094µs remote_storage=1.525µs > web_handler=583ns query_engine=1.243µs scrape=197.887µs scrape_sd=650.383µs > notify=16.671µs notify_sd=181.635µs rules=38.073939ms tracing=4.418µs > ts=2023-05-23T08:15:00.177Z caller=main.go:978 level=info msg="Server is > ready to receive web requests." ts=2023-05-23T08:15:00.177Z > caller=manager.go:974 level=info component="rule manager" msg="Starting > rule manager..." > > > Sometimes Prom will just start up again without warning and other times I > see this after all the out of bounds errors: > > ts=2023-05-23T08:18:58.284Z caller=main.go:828 level=warn msg="Received > SIGTERM, exiting gracefully..." ts=2023-05-23T08:18:58.284Z > caller=main.go:852 level=info msg="Stopping scrape discovery manager..." > ts=2023-05-23T08:18:58.284Z caller=main.go:866 level=info msg="Stopping > notify discovery manager..." ts=2023-05-23T08:18:58.284Z > caller=manager.go:988 level=info component="rule manager" msg="Stopping > rule manager..." ts=2023-05-23T08:18:58.284Z caller=main.go:848 level=info > msg="Scrape discovery manager stopped" ts=2023-05-23T08:18:58.284Z > caller=main.go:862 level=info msg="Notify discovery manager stopped" > > Is anyone able to assist with what is going wrong here? > > Thanks > Nick > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/ab9d7a7f-b4d8-49ee-b702-fda49a590102n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/ab9d7a7f-b4d8-49ee-b702-fda49a590102n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmr_GMqRcYVcAtpZKWBGNcKXacTpOSsuX5Tz5S%3DXU5kWJw%40mail.gmail.com.
