svilenvul opened a new issue, #7377: URL: https://github.com/apache/apisix/issues/7377
### Current Behavior We are now using APISIX in a Kubernetes setup with Helm (https://github.com/apache/apisix-helm-chart). APISIX is running as a service in the Istio Service Mesh with Envoy sidecar applied on it. We faced an issue where after we enabled mTLS with Istio, requests targeted to APISIX failed. During debugging we saw that the `authority` header for the outgoing requests from the APISIX was always set to `apisix_backend`. We think that his is confusing Istio during the mTLS and results in the request failure. ### Expected Behavior Requests should be successful both with Istio mTLS enabled and disabled. ### Error Logs Request Headers Info (from client grpcurl) ``` authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eyJleHAiOjE2NTY2ODg4NzUsImlhdCI6MTY1NjYwMjQ3NSwiYXV0aF90aW1lIjoxNjU2NDIwNzAzLCJqdGkiOiI5ZTk1ZDdhZS0yNzlmLTRlNTktODE0Yi1mYzNkMzNmYmM4MDEiLCJpc3MiOiJodHRwczovL3RvZ2dpZC50b2dnLmNsb3VkL2F1dGgvcmVhbG1zL3RvZ2dpZCIsInN1YiI6IjgwZTkxMzYwLWVmOTAtNDI5OC05ODRkLTcxNDBiNDY5NTFlMyIsInR5cCI6IkJlYXJlciIsImF6cCI6InN1cGVyLWFwcCIsInNlc3Npb25fc3RhdGUiOiI5ZDUwNmFjMS1mNGZlLTRiOTUtYjJkNy1iYTdjOTg0ODc1MjUiLCJhY3IiOiIwIiwic2NvcGUiOiJvcGVuaWQgZW1haWwgcHJvZmlsZSIsInNpZCI6IjlkNTA2YWMxLWY0ZmUtNGI5NS1iMmQ3LWJhN2M5ODQ4NzUyNSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiVGVzdCBUZXN0IiwicHJlZmVycmVkX3VzZXJuYW1lIjoidGVzdEB0b2dnLmNsb3VkIiwiZ2l2ZW5fbmFtZSI6IlRlc3QiLCJmYW1pbHlfbmFtZSI6IlRlc3QiLCJlbWFpbCI6InRlc3RAdG9nZy5jbG91ZCJ9.BK-EGfdyi7DoIQTRYxUFBK54f4g2IyAK6DlQDinDldjf2OXFRyWIK9OwN7Q5-hW5BO0hn0huJ4aQZ59WGUdZ5RjZqVCV3-w2ybr7BXHkwKJYnjrB0lcFy4in1WB_eiD4TMBdqb7vG6dxC8bGdm8YmBfFvJ7Ufghle33pjj67k8SJj3zUFRBK-f4umKesakfTlhlMMd ALbCTxV9jIoXPtDpvDEF6V89N7LKnnoV8Q3lPBF56PGeBokdqEJLfsb5ZQcaMeW8Fi38adqZTa8A4WefoRRsOrgEhXMYoU8DrY1EWvatgms4vJKag6bygkp_2nsNKT__hoYIDBvvJMke60VQ ``` Response headers (from client grpcurl) ``` content-length: 0 content-type: application/grpc date: Thu, 30 Jun 2022 15:39:53 GMT server: istio-envoy x-envoy-upstream-service-time: 84 ``` Logs from Envoy Proxy sidecar container for APISIX ``` { "duration"1, "downstream_remote_address":"172.20.50.105:0", "upstream_service_time":null, "upstream_local_address":"10.234.106.162:50972", "response_code_details":"upstream_reset_before_response_started{connection_termination}", "upstream_transport_failure_reason":null, "route_name":"allow_any", "response_code":200, "upstream_host":"10.234.29.1:80","user_agent":"grpcurl/v1.8.1 grpc-go/1.37.0", "downstream_local_address":"10.234.29.1:80", "x_forwarded_for":"172.20.50.105", "connection_termination_details":null, "protocol":"HTTP/2", "upstream_cluster":"PassthroughCluster", "authority":"apisix_backend", "method":"POST", "start_time":"2022-06-30T15:39:53.730Z", "path":"/xxxx.yyyy.ms.profile.ProfileService/GetUserProfile", "bytes_received":51, "response_flags":"UC", "request_id":"7e4abc76-27f4-4f40-b663-056c648608b7", "bytes_sent":0, "requested_server_name":null } ``` Logs from APISIX container ``` 127.0.0.6 - - [30/Jun/2022:15:39:53 +0000] xxxxx-api-gateway.xxxx.cloud:9443 "POST /xxxx.yyyy.ms.profile.ProfileService/GetUserProfile HTTP/2.0" 200 0 0.005 "-" "grpcurl/v1.8.1 grpc-go/1.37.0" 10.234.29.1:80 200 0.004 "grpc://xxxxxx-api-gateway.xxxx.cloud:9443" ``` ### Steps to Reproduce 1. Install Istio in k8s cluster 2. Enable Istio Strict mTLS ``` $ kubectl apply -n istio-system -f - <<EOF apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: "default" spec: mtls: mode: STRICT EOF ``` 3. Create namespace for APISIX and enable auto injection for Istio 4. Install APISIX with a Helm chart 5. Install a gRPC Service in the same namespace 6. Create Route and Upstream in APISIX - Route ``` { "uris": [ "/xxxx.yyyyy.ms.profile.ProfileService/*" ], "name": "Profile_Service", "methods": [ "POST", "GET" ], "upstream_id": "369132756648592144", "status": 1 } ``` - Upstream ``` { "nodes": [ { "host": "grpc-service.namespace.svc.cluster.local", "port": 80, "weight": 1 } ], "timeout": { "connect": 6, "read": 6, "send": 6 }, "type": "roundrobin", "scheme": "grpc", "pass_host": "pass", "name": "Profile Service" } ``` ### Environment - APISIX version: 2.12.1 - k8s version: 1.20.7 - Istio version: 1.10.3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
