Much appreciated Matt, Definetly will try your suggest , 2f5e8fb33829e7525d118acaa4a6907af95d0400 commit and I'll let you know how it goes.( in two days :D)
Thanks, Ray [email protected] 在 2020年11月26日 星期四下午4:42:26 [UTC+8] 的信中寫道： > You are using a single PersistentVolumeClaim rather than letting > Kubernetes create one per instance of the statefulset. Even though it is of > mode ReadWriteOnce, I believe it is being bounced around between the > different pods and that is causing problems. > > Rather than creating the PVC yourself, use the volumeClaimTemplates field > of the StatefulSet spec, like in this example > <https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset> > . > > /MR > > On Thu, Nov 26, 2020 at 3:48 AM 墨生 <[email protected]> wrote: > >> Hey guys, >> I'm a newbie to promethues and recently I've been working on promethues >> ha with thanos.But everything looks good after 2days running , here are >> some of the logs >> logs for prometheus-3: >> level=info ts=2020-11-26T03:45:48.626Z caller=manager.go:934 >> component="rule manager" msg="Rule manager stopped" >> level=info ts=2020-11-26T03:45:48.626Z caller=notifier.go:601 >> component=notifier msg="Stopping notification manager..." >> level=info ts=2020-11-26T03:45:48.626Z caller=main.go:789 msg="Notifier >> manager stopped" >> level=info ts=2020-11-26T03:45:48.626Z caller=main.go:615 msg="Scrape >> manager stopped" >> level=error ts=2020-11-26T03:45:48.626Z caller=main.go:798 err="opening >> storage failed: found unsequential head chunk files >> /data/chunks_head/000002 (index: 2) and /data/chunks_head/000005 (index: 5)" >> >> logs for prometheus-2: >> level=error ts=2020-11-26T03:15:20.159Z caller=db.go:730 component=tsdb >> msg="compaction failed" err="persist head block: populate block: add >> series: out-of-order series added with label set >> \"{__name__=\\\"envoy_cluster_assignment_stale\\\", >> app=\\\"istio-ingressgateway\\\", chart=\\\"gateways\\\", >> cluster_name=\\\"xds-grpc\\\", heritage=\\\"Tiller\\\", instance=\\\" >> 10.129.45.133:15090\\\", istio=\\\"ingressgateway\\\", >> job=\\\"kubernetes-pods\\\", kubernetes_namespace=\\\"istio-system\\\", >> kubernetes_pod_name=\\\"istio-ingressgateway-669cfc876b-zdcng\\\", >> pod_template_hash=\\\"669cfc876b\\\", release=\\\"istio\\\", >> service_istio_io_canonical_name=\\\"istio-ingressgateway\\\", >> service_istio_io_canonical_revision=\\\"latest\\\"}\"" >> >> My complete configurations can be found here: >> https://github.com/hairongGao/prometheus-thanos-grafana >> >> Thanks in advance, >> Ray >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/775a78b8-b4f0-494d-b1db-2454d3fb010an%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/775a78b8-b4f0-494d-b1db-2454d3fb010an%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1b8fc20f-91ff-44ae-978b-d2086d0977dcn%40googlegroups.com.
