For the benefit of the list, what was the issue and the solution? On Wednesday, 22 June 2022 at 12:54:05 UTC+1 Volker wrote:
> Hello all, > > my problem is solved. > > Thanks, > Volker > > On 6/22/22 09:25, Volker Dormeyer wrote: > > my first mail was in a ugly format... this is a resent > > > > > > Hi! > > > > I have a problem within Thanos. I know, this is a Prometheus mailing > > list... > > > > I experience following issue: The Query instance is not able to > > communicate with the Thanos Sidecar. The error message says the > > following: > > > > level=info ts=2022-06-21T13:12:13.219335188Z caller=client.go:55 > > msg="enabling client to server TLS" > > level=info ts=2022-06-21T13:12:13.219501389Z caller=options.go:115 > > msg="TLS client using provided certificate pool" > > level=info ts=2022-06-21T13:12:13.21951672Z caller=options.go:148 > > msg="TLS client authentication enabled" > > level=info ts=2022-06-21T13:12:13.223788976Z caller=options.go:31 > > protocol=gRPC msg="enabling server side TLS" > > level=info ts=2022-06-21T13:12:13.22419521Z caller=options.go:61 > > protocol=gRPC msg="server TLS client verification enabled" > > level=info ts=2022-06-21T13:12:13.224696576Z caller=query.go:705 > > msg="starting query node" > > level=info ts=2022-06-21T13:12:13.224820551Z > > caller=intrumentation.go:75 msg="changing probe status" status=healthy > > level=info ts=2022-06-21T13:12:13.224871106Z caller=http.go:73 > > service=http/server component=query msg="listening for requests and > > metrics" address=0.0.0.0:10902 > > level=info ts=2022-06-21T13:12:13.225078636Z > > caller=intrumentation.go:56 msg="changing probe status" status=ready > > level=info ts=2022-06-21T13:12:13.225222457Z caller=tls_config.go:195 > > service=http/server component=query msg="TLS is disabled." http2=false > > level=info ts=2022-06-21T13:12:13.225290133Z caller=grpc.go:131 > > service=gRPC/server component=query msg="listening for serving gRPC" > > address=0.0.0.0:10901 > > level=warn ts=2022-06-21T13:12:18.226170081Z caller=endpointset.go:517 > > component=endpointset msg="update of node failed" err="getting > > metadata: fallback fetching info from thanos-sc.dev.example.org:443: > > rpc error: code = DeadlineExceeded desc = context deadline exceeded" > > address=thanos-sc.dev.example.org:443 > > > > I already checked the certificates twice and more. What makes me > > thinking that the grpcurl result also in a error: > > > > $ grpcurl -insecure thanos-sc.local:443 list > > Failed to dial target host "thanos-sc.dev.example.org:443": remote > > error: tls: no application protocol > > > > I am able to port-forward the 10901 an them access the port > > successfully via grpcurl. > > > > My Ingress: > > > > apiVersion: networking.k8s.io/v1 > > kind: Ingress > > metadata: > > annotations: > > meta.helm.sh/release-name: prom > > meta.helm.sh/release-namespace: mon > > nginx.ingress.kubernetes.io/backend-protocol: GRPC > > nginx.ingress.kubernetes.io/ssl-redirect: "true" > > labels: > > app: kube-prometheus-stack-prometheus > > app.kubernetes.io/instance: prom > > app.kubernetes.io/managed-by: Helm > > app.kubernetes.io/part-of: kube-prometheus-stack > > app.kubernetes.io/version: 32.2.1 > > chart: kube-prometheus-stack-32.2.1 > > heritage: Helm > > release: prom > > name: prom-kube-prometheus-stack-thanos-gateway > > namespace: mon > > spec: > > ingressClassName: nginx > > rules: > > - host: thanos-sc.dev.example.org > > http: > > paths: > > - backend: > > service: > > name: prom-kube-prometheus-stack-prometheus > > port: > > number: 10901 > > path: / > > pathType: ImplementationSpecific > > tls: > > - secretName: new-tls-secret > > hosts: > > - thanos-sc.dev.example.org > > > > My Thanos Query: > > > > containers: > > - args: > > - query > > - --log.level=info > > - --log.format=logfmt > > - --grpc-address=0.0.0.0:10901 > > - --http-address=0.0.0.0:10902 > > - --query.replica-label=replica > > - --store=thanos-sc.dev.example.org:443 > > - --grpc-server-tls-cert=/certs/server/tls.crt > > - --grpc-server-tls-key=/certs/server/tls.key > > - --grpc-server-tls-client-ca=/certs/server/ca.crt > > - --grpc-client-tls-secure > > - --grpc-client-tls-cert=/certs/client/tls-cert > > - --grpc-client-tls-key=/certs/client/tls-key > > - --grpc-client-tls-ca=/certs/client/ca-cert > > - --grpc-client-server-name=thanos-sc.dev.example.org > > > > Thanks, > > Volker > > > > > > On 6/22/22 09:19, Volker Dormeyer wrote: > >> Hi! > >> > >> I have a problem within Thanos. I know, this is a Prometheus mailing > >> list... > >> > >> I experience following issue: The Query instance is not able to > >> communicate with the Thanos Sidecar. The error message says the > >> following: > >> > >> |level=info ts=2022-06-21T13:12:13.219335188Z caller=client.go:55 > >> msg="enabling client to server TLS" level=info > >> ts=2022-06-21T13:12:13.219501389Z caller=options.go:115 msg="TLS > >> client using provided certificate pool" level=info > >> ts=2022-06-21T13:12:13.21951672Z caller=options.go:148 msg="TLS > >> client authentication enabled" level=info > >> ts=2022-06-21T13:12:13.223788976Z caller=options.go:31 protocol=gRPC > >> msg="enabling server side TLS" level=info > >> ts=2022-06-21T13:12:13.22419521Z caller=options.go:61 protocol=gRPC > >> msg="server TLS client verification enabled" level=info > >> ts=2022-06-21T13:12:13.224696576Z caller=query.go:705 msg="starting > >> query node" level=info ts=2022-06-21T13:12:13.224820551Z > >> caller=intrumentation.go:75 msg="changing probe status" > >> status=healthy level=info ts=2022-06-21T13:12:13.224871106Z > >> caller=http.go:73 service=http/server component=query msg="listening > >> for requests and metrics" address=0.0.0.0:10902 level=info > >> ts=2022-06-21T13:12:13.225078636Z caller=intrumentation.go:56 > >> msg="changing probe status" status=ready level=info > >> ts=2022-06-21T13:12:13.225222457Z caller=tls_config.go:195 > >> service=http/server component=query msg="TLS is disabled." > >> http2=false level=info ts=2022-06-21T13:12:13.225290133Z > >> caller=grpc.go:131 service=gRPC/server component=query msg="listening > >> for serving gRPC" address=0.0.0.0:10901 level=warn > >> ts=2022-06-21T13:12:18.226170081Z caller=endpointset.go:517 > >> component=endpointset msg="update of node failed" err="getting > >> metadata: fallback fetching info from thanos-sc.dev.example.org:443: > >> rpc error: code = DeadlineExceeded desc = context deadline exceeded" > >> address=thanos-sc.dev.example.org:443 | > >> > >> I already checked the certificates twice and more. What makes me > >> thinking that the grpcurl result also in a error: > >> > >> |$ grpcurl -insecure thanos-sc.local:443 list Failed to dial target > >> host "thanos-sc.dev.example.org:443": remote error: tls: no > >> application protocol I am able to port-forward the 10901 an them > >> access the port successfully via grpcurl. |*My Ingress:* > >> > >> |apiVersion: networking.k8s.io/v1 kind: Ingress metadata: > >> annotations: meta.helm.sh/release-name: prom > >> meta.helm.sh/release-namespace: mon > >> nginx.ingress.kubernetes.io/backend-protocol: GRPC > >> nginx.ingress.kubernetes.io/ssl-redirect: "true" labels: app: > >> kube-prometheus-stack-prometheus app.kubernetes.io/instance: prom > >> app.kubernetes.io/managed-by: Helm app.kubernetes.io/part-of: > >> kube-prometheus-stack app.kubernetes.io/version: 32.2.1 chart: > >> kube-prometheus-stack-32.2.1 heritage: Helm release: prom name: > >> prom-kube-prometheus-stack-thanos-gateway namespace: mon spec: > >> ingressClassName: nginx rules: - host: thanos-sc.dev.example.org > >> http: paths: - backend: service: name: > >> prom-kube-prometheus-stack-prometheus port: number: 10901 path: / > >> pathType: ImplementationSpecific tls: - secretName: new-tls-secret > >> hosts: - thanos-sc.dev.example.org | > >> > >> *Thanos Query:* > >> > >> || > >> > >> containers: > >> - args: > >> - query > >> - --log.level=info > >> - --log.format=logfmt > >> - --grpc-address=0.0.0.0:10901 > >> - --http-address=0.0.0.0:10902 > >> - --query.replica-label=replica > >> - --store=thanos-sc.dev.example.org:443 > >> - --grpc-server-tls-cert=/certs/server/tls.crt > >> - --grpc-server-tls-key=/certs/server/tls.key > >> - --grpc-server-tls-client-ca=/certs/server/ca.crt > >> - --grpc-client-tls-secure > >> - --grpc-client-tls-cert=/certs/client/tls-cert > >> - --grpc-client-tls-key=/certs/client/tls-key > >> - --grpc-client-tls-ca=/certs/client/ca-cert > >> - --grpc-client-server-name=thanos-sc.dev.example.org > >> > >> > >> Do have some idea of it. > >> > >> Best Regards, > >> Volker > >> > >> || > >> > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bedf5439-5117-4c30-9389-53e2357e42b5n%40googlegroups.com.
