I had 2 Kubernetes clusters on Azure. Thanos Query was installed on Cluster
A along with Prometheus+sidecar via stable/prometheus-operator helm chart.
I needed to query the remote cluster's sidecar. I had to discover the
remote sidecar by ip (- --store=xx.xx.xx.xx:10901), which is a pain as the
ip changes periodically and it breaks the connection to Thanos query. For
solving this issue, I installed nginx controller. Below is my
configuration. However after I re-deployed Thanos query (- --store=
prometheus-ba1-nonprod.mydomain.com:80), the remote side was not
discovered. Any advice?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: prometheus-ingress
namespace: espr-prometheus-nonprod
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/grpc-backend: "true"
spec:
rules:
- host: prometheus-ba1-nonprod.mydomain.com
http:
paths:
- backend:
serviceName: prometheus-operated
servicePort: 10901
path: /
[zzhao05@esco-jumphost-prod-svc-vm-01-escojump000001 ~]$ kubectl get po
NAME READY STATUS
RESTARTS AGE
alertmanager-prometheus-prometheus-oper-alertmanager-0 2/2 Running
0 14h
nginx-ingress-controller-54db9cb9b5-lk99l 1/1 Running
0 14h
nginx-ingress-controller-54db9cb9b5-swpjc 1/1 Running
0 13h
nginx-ingress-default-backend-7c6cd67dbb-lbmcp 1/1 Running
0 14h
prometheus-kube-state-metrics-7844d8fc49-5sfzw 1/1 Running
0 14h
prometheus-prometheus-node-exporter-bkr5m 1/1 Running
3 8d
prometheus-prometheus-node-exporter-cxqnb 1/1 Running
3 7d15h
prometheus-prometheus-node-exporter-dj86m 1/1 Running
2 26d
prometheus-prometheus-node-exporter-mznr4 1/1 Running
1 15h
prometheus-prometheus-node-exporter-ppvsc 1/1 Running
2 26d
prometheus-prometheus-node-exporter-pspvk 1/1 Running
1 38h
prometheus-prometheus-node-exporter-rspbz 1/1 Running
2 26d
prometheus-prometheus-oper-operator-67ccf47d85-5kbtk 2/2 Running
0 14h
prometheus-prometheus-prometheus-oper-prometheus-0 4/4 Running
1 14h
[zzhao05@esco-jumphost-prod-svc-vm-01-escojump000001 ~]$ kubectl get svc
NAME TYPE CLUSTER-IP
EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None
<none> 9093/TCP,9094/TCP,9094/UDP 26d
nginx-ingress-controller LoadBalancer 10.0.85.179
xx.xx.xx.xx 80:32615/TCP,443:30561/TCP 7d5h
nginx-ingress-default-backend ClusterIP 10.0.26.35
<none> 80/TCP 7d5h
prometheus-kube-state-metrics ClusterIP 10.0.205.205
<none> 8080/TCP 26d
prometheus-operated ClusterIP None
<none> 9090/TCP,10901/TCP 26d
prometheus-prometheus-node-exporter ClusterIP 10.0.1.99
<none> 9100/TCP 26d
prometheus-prometheus-oper-alertmanager ClusterIP 10.0.153.203
<none> 9093/TCP 26d
prometheus-prometheus-oper-operator ClusterIP 10.0.226.205
<none> 8080/TCP,443/TCP 26d
prometheus-prometheus-oper-prometheus ClusterIP 10.0.119.255
<none> 9090/TCP 26d
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/efe18198-c8a6-43f1-a1a2-ddbf262d27a8o%40googlegroups.com.
