12GB of storage left, this is a raspberry pi btw On Tuesday, 2 February 2021 at 23:19:33 UTC Stuart Clark wrote:
> On 02/02/2021 23:16, Nial mccallister wrote: > > output of ls -l > > pi@Hydrogen:~ $ ls -l > /var/lib/prometheus/metrics2/01EWX0Y64BRZ749WK3AFSKJ98K > total 904 > drwxr-xr-x 2 prometheus prometheus 4096 Jan 25 15:00 chunks > -rw-r--r-- 1 prometheus prometheus 916885 Jan 25 15:00 index > -rw-r--r-- 1 prometheus prometheus 0 Jan 26 20:49 meta.json > -rw-r--r-- 1 prometheus prometheus 9 Jan 25 15:00 tombstone > > On Tuesday, 2 February 2021 at 23:13:11 UTC Nial mccallister wrote: > >> full log >> >> Feb 02 20:35:41 Hydrogen systemd[1]: Started Monitoring system and time >> series database. >> Feb 02 20:35:44 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:44.574969007Z caller=main.go:300 msg="Starting >> Prometheus" version="(version=2.7.1+ds, branch=debian/sid, >> revision=2.7.1+ds-3)" >> Feb 02 20:35:44 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:44.575740934Z caller=main.go:301 >> build_context="(go=go1.11.5, [email protected], >> date=20190210-18:10:02)" >> Feb 02 20:35:44 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:44.576023121Z caller=main.go:302 host_details="(Linux >> 5.4.83-v7+ #1379 SMP Mon Dec 14 13:08:57 GMT 2020 armv7l Hydrogen (none))" >> Feb 02 20:35:44 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:44.576268746Z caller=main.go:303 fd_limits="(soft=8192, >> hard=8192)" >> Feb 02 20:35:44 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:44.576487132Z caller=main.go:304 >> vm_limits="(soft=unlimited, hard=unlimited)" >> Feb 02 20:35:44 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:44.638680725Z caller=web.go:411 component=web >> msg="Start listening for connections" address=0.0.0.0:9090 >> Feb 02 20:35:44 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:44.638458225Z caller=main.go:618 msg="Starting TSDB ..." >> Feb 02 20:35:44 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:44.983999319Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1610344800000 maxt=1610409600000 >> ulid=01EVT8KEW2FBT7VEDA3FMQ3EW2 >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.025813746Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1610409600000 maxt=1610474400000 >> ulid=01EVW6CE38RRV1SQW1702SGGBB >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.050734892Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1610474400000 maxt=1610539200000 >> ulid=01EVY461P9CB9QY47RQ2T5EF9J >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.092900048Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1610539200000 maxt=1610604000000 >> ulid=01EW01Z3ENEW83R9PCKFBTNCXP >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.111339215Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1610604000000 maxt=1610668800000 >> ulid=01EW1ZRMHH8DDJAEBRER4D6V8B >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.154788954Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1610668800000 maxt=1610719200000 >> ulid=01EWG4YHPGPYPXWV77ZFSRPW1S >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.197002548Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1611136800000 maxt=1611187200000 >> ulid=01EWHN0THG5BTYJDV1W4FYF9YQ >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.238659475Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1611187200000 maxt=1611252000000 >> ulid=01EWKBYFK4QRQTZ4NYFPZH3ZMN >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.263393694Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1611252000000 maxt=1611316800000 >> ulid=01EWN9RAHT9J3MEJPZ0AVPQP5T >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.282775517Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1611316800000 maxt=1611381600000 >> ulid=01EWQ7HJ8FR4D4X0QNXJYJDKKX >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.301695256Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1611381600000 maxt=1611446400000 >> ulid=01EWS5B3EYP7DWCJHPMNZ48H43 >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.320386454Z caller=repair.go:48 component=tsdb >> msg="found healthy block" mint=1611446400000 maxt=1611511200000 >> ulid=01EWV34MVFTPX7Y917V0ZDHR3K >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.33257859Z caller=main.go:487 msg="Stopping scrape >> discovery manager..." >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.332825725Z caller=main.go:501 msg="Stopping notify >> discovery manager..." >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.332952913Z caller=main.go:523 msg="Stopping scrape >> manager..." >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.333081923Z caller=manager.go:737 component="rule >> manager" msg="Stopping rule manager..." >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.333049267Z caller=main.go:483 msg="Scrape discovery >> manager stopped" >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.333200413Z caller=manager.go:743 component="rule >> manager" msg="Rule manager stopped" >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.333277808Z caller=main.go:497 msg="Notify discovery >> manager stopped" >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.333333642Z caller=notifier.go:521 component=notifier >> msg="Stopping notification manager..." >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.333467236Z caller=main.go:677 msg="Notifier manager >> stopped" >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=info >> ts=2021-02-02T20:35:45.347324267Z caller=main.go:517 msg="Scrape manager >> stopped" >> Feb 02 20:35:45 Hydrogen prometheus[437]: level=error >> ts=2021-02-02T20:35:45.347679163Z caller=main.go:686 err="opening storage >> failed: block dir: >> \"/var/lib/prometheus/metrics2/01EWX0Y64BRZ749WK3AFSKJ98K\": unexpected end >> of JSON input" >> >> Did the server running Prometheus run out of storage? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7f0e4988-0a9e-4264-aff6-323526e90c5fn%40googlegroups.com.
