Hey guys,
I'm a newbie to promethues and recently I've been working on promethues ha
with thanos.But everything looks good after 2days running , here are some
of the logs
logs for prometheus-3:
level=info ts=2020-11-26T03:45:48.626Z caller=manager.go:934
component="rule manager" msg="Rule manager stopped"
level=info ts=2020-11-26T03:45:48.626Z caller=notifier.go:601
component=notifier msg="Stopping notification manager..."
level=info ts=2020-11-26T03:45:48.626Z caller=main.go:789 msg="Notifier
manager stopped"
level=info ts=2020-11-26T03:45:48.626Z caller=main.go:615 msg="Scrape
manager stopped"
level=error ts=2020-11-26T03:45:48.626Z caller=main.go:798 err="opening
storage failed: found unsequential head chunk files
/data/chunks_head/000002 (index: 2) and /data/chunks_head/000005 (index: 5)"
logs for prometheus-2:
level=error ts=2020-11-26T03:15:20.159Z caller=db.go:730 component=tsdb
msg="compaction failed" err="persist head block: populate block: add
series: out-of-order series added with label set
\"{__name__=\\\"envoy_cluster_assignment_stale\\\",
app=\\\"istio-ingressgateway\\\", chart=\\\"gateways\\\",
cluster_name=\\\"xds-grpc\\\", heritage=\\\"Tiller\\\",
instance=\\\"10.129.45.133:15090\\\", istio=\\\"ingressgateway\\\",
job=\\\"kubernetes-pods\\\", kubernetes_namespace=\\\"istio-system\\\",
kubernetes_pod_name=\\\"istio-ingressgateway-669cfc876b-zdcng\\\",
pod_template_hash=\\\"669cfc876b\\\", release=\\\"istio\\\",
service_istio_io_canonical_name=\\\"istio-ingressgateway\\\",
service_istio_io_canonical_revision=\\\"latest\\\"}\""
My complete configurations can be found here:
https://github.com/hairongGao/prometheus-thanos-grafana
Thanks in advance,
Ray
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/775a78b8-b4f0-494d-b1db-2454d3fb010an%40googlegroups.com.
