*Thanos Sidecar running on other cluster was not discovered by dnssrv on Thanos Querier. In my Thanos Querier deployment yaml file, I used “—store=dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc.cluster.local” service discover the sidecar. But It ONLY discovered on the prometheus instances that run on the same cluster with Thanos Querier. For the other prometheus instances that run on other clusters, sidecar didn’t report to Thanos Querier so that I had to hard code in the IP address and port. Any idea why DNS Service Discovery did not work on other clusters? Thanks. I was using prometheus-operator helm chart to deploy prometheus and Thanos Sidecar. The service name is exactly identical as “prometheus-operated”.*
alex@Zhangs-MacBook prometheus-operator % kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 9d prometheus-operated ClusterIP None <none> 9090/TCP,10901/TCP 9d prometheus-operator-alertmanager ClusterIP 10.0.55.23 <none> 9093/TCP 9d prometheus-operator-grafana ClusterIP 10.0.177.193 <none> 80/TCP 9d prometheus-operator-kube-state-metrics ClusterIP 10.0.238.98 <none> 8080/TCP 9d prometheus-operator-operator ClusterIP 10.0.223.147 <none> 8080/TCP,443/TCP 9d prometheus-operator-prometheus ClusterIP 10.0.127.8 <none> 9090/TCP 9d prometheus-operator-prometheus-node-exporter ClusterIP 10.0.127.50 <none> 9100/TCP 9d prometheus-prometheus-operator-prometheus-0 NodePort 10.0.172.129 <none> 9090:30720/TCP 9d thanos-bucket ClusterIP 10.0.172.148 <none> 10902/TCP 30h thanos-compactor ClusterIP 10.0.243.77 <none> 10902/TCP 6d thanos-querier ClusterIP 10.0.151.135 <none> 10901/TCP,9090/TCP 6d12h thanos-store ClusterIP None <none> 10901/TCP,10902/TCP 6d13h -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/14bfb11f-cc61-47a3-9949-010d629b8c8d%40googlegroups.com.
