Hi Tristan, Thank you for your reply and sorry for the weird background color.
For the backend, it doesn’t actually use http protocol. It uses GRPC. For your 2nd point, actually that’s what I tried. I added the annotation below when I was configuring nginx ingress controller and an internal ip to my network was assigned to the ingress service and pointed it to the sidecar service. The problem was that on Thanos Query side, the sidecar was not able to be discovered by the ingress load balancer ip. service.beta.kubernetes.io/azure-load-balancer-internal <http://service.beta.kubernetes.io/azure-load-balancer-internal>: "true" On Jul 2, 2020, at 1:21 AM, Tristan Colgate <[email protected]> wrote: > > (before I answer, just a quick note that your mail client seems to use > HTML with a black background and white foreground for text, and it > looks super weird if the reader has a white background mail client as > it is all stripey). > > Two things: > > 1. it sounds like maybe your nginx ingress isn't doing the right > thing. I've not used the nginx ingress, but you may need to indicate > to it that the backend requires http2. > > 2. Whether you point the querier at the nginx ingress, or the sidecar > pod, you probably want to do that via a kubernetes service rather than > direct to the pod's IP. If you use a service with a clusterIP, that IP > is fixed until such time as your delete that service. If you are in a > different cluster you may need to use a service with type > "loadbalancer", you may need to look at how to make that an internal > only load balancer in azure (normally done by adding an annotation), > which would restrict it to be internal to the network of your cloud > provider, and not public. > > On Thu, 2 Jul 2020 at 07:04, Zhang Zhao <[email protected]> wrote: >> >> I setup 2 Kubernetes clusters on Azure, Thanos Querier is in one cluster, >> and a local prometheus+sidecar. I need to query the remote cluster Thanos >> sidecar. I created ingress-nginx with support for grpc on remote cluster. >> But the sidecar grpc failed in Thanos Querier so that Querier was not able >> to discover the sidecar on remote cluster. I still had to use the ip and >> port to discover the sidecar on remote clusters. Everytime the pod is >> restarted, the ip changes and the connection is broken. Any advice? >> >> >> >> >> >> apiVersion: extensions/v1beta1 >> >> kind: Ingress >> >> metadata: >> >> name: thanos-sidecar-ingress >> >> namespace: espr-prometheus-nonprod >> >> annotations: >> >> kubernetes.io/ingress.class: “nginx” >> >> nginx.ingress.kubernetes.io/ssl-redirect: "false" >> >> nginx.ingress.kubernetes.io/backend-protocol: “GRPC” >> >> spec: >> >> rules: >> >> - host: xxx.example.com >> >> http: >> >> paths: >> >> - backend: >> >> serviceName: prometheus-operated >> >> servicePort: 10901 >> >> path: /(.*) >> >> >> >> Zhang >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/97dd931f-4720-4572-9c89-cccb3d16ccf8o%40googlegroups.com. > > > > -- > Tristan Colgate-McFarlane > ---- > "You can get all your daily vitamins from 52 pints of guiness, and a > glass of milk" -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/FD0F1071-C0BC-475D-97A2-DF77D1663AD3%40gmail.com.
