my first mail was in a ugly format... this is a resent
Hi!
I have a problem within Thanos. I know, this is a Prometheus mailing
list...
I experience following issue: The Query instance is not able to
communicate with the Thanos Sidecar. The error message says the following:
level=info ts=2022-06-21T13:12:13.219335188Z caller=client.go:55
msg="enabling client to server TLS"
level=info ts=2022-06-21T13:12:13.219501389Z caller=options.go:115
msg="TLS client using provided certificate pool"
level=info ts=2022-06-21T13:12:13.21951672Z caller=options.go:148
msg="TLS client authentication enabled"
level=info ts=2022-06-21T13:12:13.223788976Z caller=options.go:31
protocol=gRPC msg="enabling server side TLS"
level=info ts=2022-06-21T13:12:13.22419521Z caller=options.go:61
protocol=gRPC msg="server TLS client verification enabled"
level=info ts=2022-06-21T13:12:13.224696576Z caller=query.go:705
msg="starting query node"
level=info ts=2022-06-21T13:12:13.224820551Z caller=intrumentation.go:75
msg="changing probe status" status=healthy
level=info ts=2022-06-21T13:12:13.224871106Z caller=http.go:73
service=http/server component=query msg="listening for requests and
metrics" address=0.0.0.0:10902
level=info ts=2022-06-21T13:12:13.225078636Z caller=intrumentation.go:56
msg="changing probe status" status=ready
level=info ts=2022-06-21T13:12:13.225222457Z caller=tls_config.go:195
service=http/server component=query msg="TLS is disabled." http2=false
level=info ts=2022-06-21T13:12:13.225290133Z caller=grpc.go:131
service=gRPC/server component=query msg="listening for serving gRPC"
address=0.0.0.0:10901
level=warn ts=2022-06-21T13:12:18.226170081Z caller=endpointset.go:517
component=endpointset msg="update of node failed" err="getting metadata:
fallback fetching info from thanos-sc.dev.example.org:443: rpc error:
code = DeadlineExceeded desc = context deadline exceeded"
address=thanos-sc.dev.example.org:443
I already checked the certificates twice and more. What makes me
thinking that the grpcurl result also in a error:
$ grpcurl -insecure thanos-sc.local:443 list
Failed to dial target host "thanos-sc.dev.example.org:443": remote
error: tls: no application protocol
I am able to port-forward the 10901 an them access the port successfully via
grpcurl.
My Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
meta.helm.sh/release-name: prom
meta.helm.sh/release-namespace: mon
nginx.ingress.kubernetes.io/backend-protocol: GRPC
nginx.ingress.kubernetes.io/ssl-redirect: "true"
labels:
app: kube-prometheus-stack-prometheus
app.kubernetes.io/instance: prom
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: kube-prometheus-stack
app.kubernetes.io/version: 32.2.1
chart: kube-prometheus-stack-32.2.1
heritage: Helm
release: prom
name: prom-kube-prometheus-stack-thanos-gateway
namespace: mon
spec:
ingressClassName: nginx
rules:
- host: thanos-sc.dev.example.org
http:
paths:
- backend:
service:
name: prom-kube-prometheus-stack-prometheus
port:
number: 10901
path: /
pathType: ImplementationSpecific
tls:
- secretName: new-tls-secret
hosts:
- thanos-sc.dev.example.org
My Thanos Query:
containers:
- args:
- query
- --log.level=info
- --log.format=logfmt
- --grpc-address=0.0.0.0:10901
- --http-address=0.0.0.0:10902
- --query.replica-label=replica
- --store=thanos-sc.dev.example.org:443
- --grpc-server-tls-cert=/certs/server/tls.crt
- --grpc-server-tls-key=/certs/server/tls.key
- --grpc-server-tls-client-ca=/certs/server/ca.crt
- --grpc-client-tls-secure
- --grpc-client-tls-cert=/certs/client/tls-cert
- --grpc-client-tls-key=/certs/client/tls-key
- --grpc-client-tls-ca=/certs/client/ca-cert
- --grpc-client-server-name=thanos-sc.dev.example.org
Thanks,
Volker
On 6/22/22 09:19, Volker Dormeyer wrote:
Hi!
I have a problem within Thanos. I know, this is a Prometheus mailing
list...
I experience following issue: The Query instance is not able to
communicate with the Thanos Sidecar. The error message says the
following:
|level=info ts=2022-06-21T13:12:13.219335188Z caller=client.go:55
msg="enabling client to server TLS" level=info
ts=2022-06-21T13:12:13.219501389Z caller=options.go:115 msg="TLS
client using provided certificate pool" level=info
ts=2022-06-21T13:12:13.21951672Z caller=options.go:148 msg="TLS client
authentication enabled" level=info ts=2022-06-21T13:12:13.223788976Z
caller=options.go:31 protocol=gRPC msg="enabling server side TLS"
level=info ts=2022-06-21T13:12:13.22419521Z caller=options.go:61
protocol=gRPC msg="server TLS client verification enabled" level=info
ts=2022-06-21T13:12:13.224696576Z caller=query.go:705 msg="starting
query node" level=info ts=2022-06-21T13:12:13.224820551Z
caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2022-06-21T13:12:13.224871106Z caller=http.go:73
service=http/server component=query msg="listening for requests and
metrics" address=0.0.0.0:10902 level=info
ts=2022-06-21T13:12:13.225078636Z caller=intrumentation.go:56
msg="changing probe status" status=ready level=info
ts=2022-06-21T13:12:13.225222457Z caller=tls_config.go:195
service=http/server component=query msg="TLS is disabled." http2=false
level=info ts=2022-06-21T13:12:13.225290133Z caller=grpc.go:131
service=gRPC/server component=query msg="listening for serving gRPC"
address=0.0.0.0:10901 level=warn ts=2022-06-21T13:12:18.226170081Z
caller=endpointset.go:517 component=endpointset msg="update of node
failed" err="getting metadata: fallback fetching info from
thanos-sc.dev.example.org:443: rpc error: code = DeadlineExceeded desc
= context deadline exceeded" address=thanos-sc.dev.example.org:443 |
I already checked the certificates twice and more. What makes me
thinking that the grpcurl result also in a error:
|$ grpcurl -insecure thanos-sc.local:443 list Failed to dial target
host "thanos-sc.dev.example.org:443": remote error: tls: no
application protocol I am able to port-forward the 10901 an them
access the port successfully via grpcurl. |*My Ingress:*
|apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations:
meta.helm.sh/release-name: prom meta.helm.sh/release-namespace: mon
nginx.ingress.kubernetes.io/backend-protocol: GRPC
nginx.ingress.kubernetes.io/ssl-redirect: "true" labels: app:
kube-prometheus-stack-prometheus app.kubernetes.io/instance: prom
app.kubernetes.io/managed-by: Helm app.kubernetes.io/part-of:
kube-prometheus-stack app.kubernetes.io/version: 32.2.1 chart:
kube-prometheus-stack-32.2.1 heritage: Helm release: prom name:
prom-kube-prometheus-stack-thanos-gateway namespace: mon spec:
ingressClassName: nginx rules: - host: thanos-sc.dev.example.org http:
paths: - backend: service: name: prom-kube-prometheus-stack-prometheus
port: number: 10901 path: / pathType: ImplementationSpecific tls: -
secretName: new-tls-secret hosts: - thanos-sc.dev.example.org |
*Thanos Query:*
||
containers:
- args:
- query
- --log.level=info
- --log.format=logfmt
- --grpc-address=0.0.0.0:10901
- --http-address=0.0.0.0:10902
- --query.replica-label=replica
- --store=thanos-sc.dev.example.org:443
- --grpc-server-tls-cert=/certs/server/tls.crt
- --grpc-server-tls-key=/certs/server/tls.key
- --grpc-server-tls-client-ca=/certs/server/ca.crt
- --grpc-client-tls-secure
- --grpc-client-tls-cert=/certs/client/tls-cert
- --grpc-client-tls-key=/certs/client/tls-key
- --grpc-client-tls-ca=/certs/client/ca-cert
- --grpc-client-server-name=thanos-sc.dev.example.org
Do have some idea of it.
Best Regards,
Volker
||
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bedc180c-3f97-8040-2d91-58f1fb0f357d%40ixolution.de.
