On 02/02/2021 23:16, Nial mccallister wrote:
output of ls -l
pi@Hydrogen:~ $ ls -l
/var/lib/prometheus/metrics2/01EWX0Y64BRZ749WK3AFSKJ98K
total 904
drwxr-xr-x 2 prometheus prometheus 4096 Jan 25 15:00 chunks
-rw-r--r-- 1 prometheus prometheus 916885 Jan 25 15:00 index
-rw-r--r-- 1 prometheus prometheus 0 Jan 26 20:49 meta.json
-rw-r--r-- 1 prometheus prometheus 9 Jan 25 15:00 tombstone
On Tuesday, 2 February 2021 at 23:13:11 UTC Nial mccallister wrote:
full log
Feb 02 20:35:41 Hydrogen systemd[1]: Started Monitoring system and
time series database.
Feb 02 20:35:44 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:44.574969007Z caller=main.go:300 msg="Starting
Prometheus" version="(version=2.7.1+ds, branch=debian/sid,
revision=2.7.1+ds-3)"
Feb 02 20:35:44 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:44.575740934Z caller=main.go:301
build_context="(go=go1.11.5,
[email protected], date=20190210-18:10:02)"
Feb 02 20:35:44 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:44.576023121Z caller=main.go:302
host_details="(Linux 5.4.83-v7+ #1379 SMP Mon Dec 14 13:08:57 GMT
2020 armv7l Hydrogen (none))"
Feb 02 20:35:44 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:44.576268746Z caller=main.go:303
fd_limits="(soft=8192, hard=8192)"
Feb 02 20:35:44 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:44.576487132Z caller=main.go:304
vm_limits="(soft=unlimited, hard=unlimited)"
Feb 02 20:35:44 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:44.638680725Z caller=web.go:411 component=web
msg="Start listening for connections" address=0.0.0.0:9090
<http://0.0.0.0:9090>
Feb 02 20:35:44 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:44.638458225Z caller=main.go:618 msg="Starting
TSDB ..."
Feb 02 20:35:44 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:44.983999319Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1610344800000
maxt=1610409600000 ulid=01EVT8KEW2FBT7VEDA3FMQ3EW2
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.025813746Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1610409600000
maxt=1610474400000 ulid=01EVW6CE38RRV1SQW1702SGGBB
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.050734892Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1610474400000
maxt=1610539200000 ulid=01EVY461P9CB9QY47RQ2T5EF9J
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.092900048Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1610539200000
maxt=1610604000000 ulid=01EW01Z3ENEW83R9PCKFBTNCXP
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.111339215Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1610604000000
maxt=1610668800000 ulid=01EW1ZRMHH8DDJAEBRER4D6V8B
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.154788954Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1610668800000
maxt=1610719200000 ulid=01EWG4YHPGPYPXWV77ZFSRPW1S
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.197002548Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1611136800000
maxt=1611187200000 ulid=01EWHN0THG5BTYJDV1W4FYF9YQ
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.238659475Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1611187200000
maxt=1611252000000 ulid=01EWKBYFK4QRQTZ4NYFPZH3ZMN
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.263393694Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1611252000000
maxt=1611316800000 ulid=01EWN9RAHT9J3MEJPZ0AVPQP5T
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.282775517Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1611316800000
maxt=1611381600000 ulid=01EWQ7HJ8FR4D4X0QNXJYJDKKX
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.301695256Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1611381600000
maxt=1611446400000 ulid=01EWS5B3EYP7DWCJHPMNZ48H43
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.320386454Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1611446400000
maxt=1611511200000 ulid=01EWV34MVFTPX7Y917V0ZDHR3K
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.33257859Z caller=main.go:487 msg="Stopping
scrape discovery manager..."
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.332825725Z caller=main.go:501 msg="Stopping
notify discovery manager..."
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.332952913Z caller=main.go:523 msg="Stopping
scrape manager..."
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.333081923Z caller=manager.go:737
component="rule manager" msg="Stopping rule manager..."
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.333049267Z caller=main.go:483 msg="Scrape
discovery manager stopped"
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.333200413Z caller=manager.go:743
component="rule manager" msg="Rule manager stopped"
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.333277808Z caller=main.go:497 msg="Notify
discovery manager stopped"
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.333333642Z caller=notifier.go:521
component=notifier msg="Stopping notification manager..."
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.333467236Z caller=main.go:677 msg="Notifier
manager stopped"
Feb 02 20:35:45 Hydrogen prometheus[437]: level=info
ts=2021-02-02T20:35:45.347324267Z caller=main.go:517 msg="Scrape
manager stopped"
Feb 02 20:35:45 Hydrogen prometheus[437]: level=error
ts=2021-02-02T20:35:45.347679163Z caller=main.go:686 err="opening
storage failed: block dir:
\"/var/lib/prometheus/metrics2/01EWX0Y64BRZ749WK3AFSKJ98K\":
unexpected end of JSON input"
Did the server running Prometheus run out of storage?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/f862d7d7-13df-7c1d-bb7e-f8da106707dd%40Jahingo.com.
