grpc uses http/2, some load balancers need to be explicitly configured, and most require http/2 over Https. You'll need to setup a DNS entry for the loadbalancer IP, or use the IP directly. What options are you providing to query ?
On Thu, 2 Jul 2020, 17:56 Zhang Zhao, <[email protected]> wrote: > Hi Tristan, > Thank you for your reply and sorry for the weird background color. > > For the backend, it doesn’t actually use http protocol. It uses GRPC. For > your 2nd point, actually that’s what I tried. I added the annotation below > when I was configuring nginx ingress controller and an internal ip to my > network was assigned to the ingress service and pointed it to the sidecar > service. The problem was that on Thanos Query side, the sidecar was not > able to be discovered by the ingress load balancer ip. > > service.beta.kubernetes.io/azure-load-balancer-internal: "true" > > > > > > > On Jul 2, 2020, at 1:21 AM, Tristan Colgate <[email protected]> wrote: > > > (before I answer, just a quick note that your mail client seems to use > HTML with a black background and white foreground for text, and it > looks super weird if the reader has a white background mail client as > it is all stripey). > > Two things: > > 1. it sounds like maybe your nginx ingress isn't doing the right > thing. I've not used the nginx ingress, but you may need to indicate > to it that the backend requires http2. > > 2. Whether you point the querier at the nginx ingress, or the sidecar > pod, you probably want to do that via a kubernetes service rather than > direct to the pod's IP. If you use a service with a clusterIP, that IP > is fixed until such time as your delete that service. If you are in a > different cluster you may need to use a service with type > "loadbalancer", you may need to look at how to make that an internal > only load balancer in azure (normally done by adding an annotation), > which would restrict it to be internal to the network of your cloud > provider, and not public. > > On Thu, 2 Jul 2020 at 07:04, Zhang Zhao <[email protected]> wrote: > > > I setup 2 Kubernetes clusters on Azure, Thanos Querier is in one cluster, > and a local prometheus+sidecar. I need to query the remote cluster Thanos > sidecar. I created ingress-nginx with support for grpc on remote cluster. > But the sidecar grpc failed in Thanos Querier so that Querier was not able > to discover the sidecar on remote cluster. I still had to use the ip and > port to discover the sidecar on remote clusters. Everytime the pod is > restarted, the ip changes and the connection is broken. Any advice? > > > > > > apiVersion: extensions/v1beta1 > > kind: Ingress > > metadata: > > name: thanos-sidecar-ingress > > namespace: espr-prometheus-nonprod > > annotations: > > kubernetes.io/ingress.class: “nginx” > > nginx.ingress.kubernetes.io/ssl-redirect: "false" > > nginx.ingress.kubernetes.io/backend-protocol: “GRPC” > > spec: > > rules: > > - host: xxx.example.com > > http: > > paths: > > - backend: > > serviceName: prometheus-operated > > servicePort: 10901 > > path: /(.*) > > > > Zhang > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/97dd931f-4720-4572-9c89-cccb3d16ccf8o%40googlegroups.com > . > > > > > -- > Tristan Colgate-McFarlane > ---- > "You can get all your daily vitamins from 52 pints of guiness, and a > glass of milk" > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPGZSGKoEbiis%3DfQ7PiGwsRko4G8B3GoO2TZ8CHu4fEUB9fNhw%40mail.gmail.com.
