Unfortunatly the problem does still exist. :( When I wrote my last Mail,
I was in opinion that the problem is solved.
Meanwhile I use an Envoy proxy to have a better TLS handling. Now the same
problem exist between the envoy forward proxy and the Ingress in the other
Cluster.
On 6/22/22 14:17, Brian Candler wrote:
For the benefit of the list, what was the issue and the solution?
On Wednesday, 22 June 2022 at 12:54:05 UTC+1 Volker wrote:
Hello all,
my problem is solved.
Thanks,
Volker
On 6/22/22 09:25, Volker Dormeyer wrote:
> my first mail was in a ugly format... this is a resent
>
>
> Hi!
>
> I have a problem within Thanos. I know, this is a Prometheus
mailing
> list...
>
> I experience following issue: The Query instance is not able to
> communicate with the Thanos Sidecar. The error message says the
> following:
>
> level=info ts=2022-06-21T13:12:13.219335188Z caller=client.go:55
> msg="enabling client to server TLS"
> level=info ts=2022-06-21T13:12:13.219501389Z caller=options.go:115
> msg="TLS client using provided certificate pool"
> level=info ts=2022-06-21T13:12:13.21951672Z caller=options.go:148
> msg="TLS client authentication enabled"
> level=info ts=2022-06-21T13:12:13.223788976Z caller=options.go:31
> protocol=gRPC msg="enabling server side TLS"
> level=info ts=2022-06-21T13:12:13.22419521Z caller=options.go:61
> protocol=gRPC msg="server TLS client verification enabled"
> level=info ts=2022-06-21T13:12:13.224696576Z caller=query.go:705
> msg="starting query node"
> level=info ts=2022-06-21T13:12:13.224820551Z
> caller=intrumentation.go:75 msg="changing probe status"
status=healthy
> level=info ts=2022-06-21T13:12:13.224871106Z caller=http.go:73
> service=http/server component=query msg="listening for requests and
> metrics" address=0.0.0.0:10902 <http://0.0.0.0:10902>
> level=info ts=2022-06-21T13:12:13.225078636Z
> caller=intrumentation.go:56 msg="changing probe status"
status=ready
> level=info ts=2022-06-21T13:12:13.225222457Z
caller=tls_config.go:195
> service=http/server component=query msg="TLS is disabled."
http2=false
> level=info ts=2022-06-21T13:12:13.225290133Z caller=grpc.go:131
> service=gRPC/server component=query msg="listening for serving
gRPC"
> address=0.0.0.0:10901 <http://0.0.0.0:10901>
> level=warn ts=2022-06-21T13:12:18.226170081Z
caller=endpointset.go:517
> component=endpointset msg="update of node failed" err="getting
> metadata: fallback fetching info from
thanos-sc.dev.example.org:443 <http://thanos-sc.dev.example.org:443>:
> rpc error: code = DeadlineExceeded desc = context deadline
exceeded"
> address=thanos-sc.dev.example.org:443
<http://thanos-sc.dev.example.org:443>
>
> I already checked the certificates twice and more. What makes me
> thinking that the grpcurl result also in a error:
>
> $ grpcurl -insecure thanos-sc.local:443 list
> Failed to dial target host "thanos-sc.dev.example.org:443
<http://thanos-sc.dev.example.org:443>": remote
> error: tls: no application protocol
>
> I am able to port-forward the 10901 an them access the port
> successfully via grpcurl.
>
> My Ingress:
>
> apiVersion: networking.k8s.io/v1 <http://networking.k8s.io/v1>
> kind: Ingress
> metadata:
> annotations:
> meta.helm.sh/release-name <http://meta.helm.sh/release-name>: prom
> meta.helm.sh/release-namespace
<http://meta.helm.sh/release-namespace>: mon
> nginx.ingress.kubernetes.io/backend-protocol
<http://nginx.ingress.kubernetes.io/backend-protocol>: GRPC
> nginx.ingress.kubernetes.io/ssl-redirect
<http://nginx.ingress.kubernetes.io/ssl-redirect>: "true"
> labels:
> app: kube-prometheus-stack-prometheus
> app.kubernetes.io/instance <http://app.kubernetes.io/instance>:
prom
> app.kubernetes.io/managed-by
<http://app.kubernetes.io/managed-by>: Helm
> app.kubernetes.io/part-of <http://app.kubernetes.io/part-of>:
kube-prometheus-stack
> app.kubernetes.io/version <http://app.kubernetes.io/version>:
32.2.1
> chart: kube-prometheus-stack-32.2.1
> heritage: Helm
> release: prom
> name: prom-kube-prometheus-stack-thanos-gateway
> namespace: mon
> spec:
> ingressClassName: nginx
> rules:
> - host: thanos-sc.dev.example.org
<http://thanos-sc.dev.example.org>
> http:
> paths:
> - backend:
> service:
> name: prom-kube-prometheus-stack-prometheus
> port:
> number: 10901
> path: /
> pathType: ImplementationSpecific
> tls:
> - secretName: new-tls-secret
> hosts:
> - thanos-sc.dev.example.org
<http://thanos-sc.dev.example.org>
>
> My Thanos Query:
>
> containers:
> - args:
> - query
> - --log.level=info
> - --log.format=logfmt
> - --grpc-address=0.0.0.0:10901 <http://0.0.0.0:10901>
> - --http-address=0.0.0.0:10902 <http://0.0.0.0:10902>
> - --query.replica-label=replica
> - --store=thanos-sc.dev.example.org:443
<http://thanos-sc.dev.example.org:443>
> - --grpc-server-tls-cert=/certs/server/tls.crt
> - --grpc-server-tls-key=/certs/server/tls.key
> - --grpc-server-tls-client-ca=/certs/server/ca.crt
> - --grpc-client-tls-secure
> - --grpc-client-tls-cert=/certs/client/tls-cert
> - --grpc-client-tls-key=/certs/client/tls-key
> - --grpc-client-tls-ca=/certs/client/ca-cert
> - --grpc-client-server-name=thanos-sc.dev.example.org
<http://thanos-sc.dev.example.org>
>
> Thanks,
> Volker
>
>
> On 6/22/22 09:19, Volker Dormeyer wrote:
>> Hi!
>>
>> I have a problem within Thanos. I know, this is a Prometheus
mailing
>> list...
>>
>> I experience following issue: The Query instance is not able to
>> communicate with the Thanos Sidecar. The error message says the
>> following:
>>
>> |level=info ts=2022-06-21T13:12:13.219335188Z caller=client.go:55
>> msg="enabling client to server TLS" level=info
>> ts=2022-06-21T13:12:13.219501389Z caller=options.go:115 msg="TLS
>> client using provided certificate pool" level=info
>> ts=2022-06-21T13:12:13.21951672Z caller=options.go:148 msg="TLS
>> client authentication enabled" level=info
>> ts=2022-06-21T13:12:13.223788976Z caller=options.go:31
protocol=gRPC
>> msg="enabling server side TLS" level=info
>> ts=2022-06-21T13:12:13.22419521Z caller=options.go:61
protocol=gRPC
>> msg="server TLS client verification enabled" level=info
>> ts=2022-06-21T13:12:13.224696576Z caller=query.go:705
msg="starting
>> query node" level=info ts=2022-06-21T13:12:13.224820551Z
>> caller=intrumentation.go:75 msg="changing probe status"
>> status=healthy level=info ts=2022-06-21T13:12:13.224871106Z
>> caller=http.go:73 service=http/server component=query
msg="listening
>> for requests and metrics" address=0.0.0.0:10902
<http://0.0.0.0:10902> level=info
>> ts=2022-06-21T13:12:13.225078636Z caller=intrumentation.go:56
>> msg="changing probe status" status=ready level=info
>> ts=2022-06-21T13:12:13.225222457Z caller=tls_config.go:195
>> service=http/server component=query msg="TLS is disabled."
>> http2=false level=info ts=2022-06-21T13:12:13.225290133Z
>> caller=grpc.go:131 service=gRPC/server component=query
msg="listening
>> for serving gRPC" address=0.0.0.0:10901 <http://0.0.0.0:10901>
level=warn
>> ts=2022-06-21T13:12:18.226170081Z caller=endpointset.go:517
>> component=endpointset msg="update of node failed" err="getting
>> metadata: fallback fetching info from
thanos-sc.dev.example.org:443 <http://thanos-sc.dev.example.org:443>:
>> rpc error: code = DeadlineExceeded desc = context deadline
exceeded"
>> address=thanos-sc.dev.example.org:443
<http://thanos-sc.dev.example.org:443> |
>>
>> I already checked the certificates twice and more. What makes me
>> thinking that the grpcurl result also in a error:
>>
>> |$ grpcurl -insecure thanos-sc.local:443 list Failed to dial
target
>> host "thanos-sc.dev.example.org:443
<http://thanos-sc.dev.example.org:443>": remote error: tls: no
>> application protocol I am able to port-forward the 10901 an them
>> access the port successfully via grpcurl. |*My Ingress:*
>>
>> |apiVersion: networking.k8s.io/v1 <http://networking.k8s.io/v1>
kind: Ingress metadata:
>> annotations: meta.helm.sh/release-name
<http://meta.helm.sh/release-name>: prom
>> meta.helm.sh/release-namespace
<http://meta.helm.sh/release-namespace>: mon
>> nginx.ingress.kubernetes.io/backend-protocol
<http://nginx.ingress.kubernetes.io/backend-protocol>: GRPC
>> nginx.ingress.kubernetes.io/ssl-redirect
<http://nginx.ingress.kubernetes.io/ssl-redirect>: "true" labels:
app:
>> kube-prometheus-stack-prometheus app.kubernetes.io/instance
<http://app.kubernetes.io/instance>: prom
>> app.kubernetes.io/managed-by
<http://app.kubernetes.io/managed-by>: Helm
app.kubernetes.io/part-of <http://app.kubernetes.io/part-of>:
>> kube-prometheus-stack app.kubernetes.io/version
<http://app.kubernetes.io/version>: 32.2.1 chart:
>> kube-prometheus-stack-32.2.1 heritage: Helm release: prom name:
>> prom-kube-prometheus-stack-thanos-gateway namespace: mon spec:
>> ingressClassName: nginx rules: - host:
thanos-sc.dev.example.org <http://thanos-sc.dev.example.org>
>> http: paths: - backend: service: name:
>> prom-kube-prometheus-stack-prometheus port: number: 10901 path: /
>> pathType: ImplementationSpecific tls: - secretName: new-tls-secret
>> hosts: - thanos-sc.dev.example.org
<http://thanos-sc.dev.example.org> |
>>
>> *Thanos Query:*
>>
>> ||
>>
>> containers:
>> - args:
>> - query
>> - --log.level=info
>> - --log.format=logfmt
>> - --grpc-address=0.0.0.0:10901 <http://0.0.0.0:10901>
>> - --http-address=0.0.0.0:10902 <http://0.0.0.0:10902>
>> - --query.replica-label=replica
>> - --store=thanos-sc.dev.example.org:443
<http://thanos-sc.dev.example.org:443>
>> - --grpc-server-tls-cert=/certs/server/tls.crt
>> - --grpc-server-tls-key=/certs/server/tls.key
>> - --grpc-server-tls-client-ca=/certs/server/ca.crt
>> - --grpc-client-tls-secure
>> - --grpc-client-tls-cert=/certs/client/tls-cert
>> - --grpc-client-tls-key=/certs/client/tls-key
>> - --grpc-client-tls-ca=/certs/client/ca-cert
>> - --grpc-client-server-name=thanos-sc.dev.example.org
<http://thanos-sc.dev.example.org>
>>
>>
>> Do have some idea of it.
>>
>> Best Regards,
>> Volker
>>
>> ||
>>
>
--
You received this message because you are subscribed to the Google
Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bedf5439-5117-4c30-9389-53e2357e42b5n%40googlegroups.com
<https://groups.google.com/d/msgid/prometheus-users/bedf5439-5117-4c30-9389-53e2357e42b5n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/cd79517f-ea1b-1a40-8a15-d02bfc5e4626%40ixolution.de.
