FlyTOmeLight opened a new issue, #9240: URL: https://github.com/apache/apisix/issues/9240
### Current Behavior I am using apisix to proxy my grpc service, and I found the following issue while doing benchmark. This is the access log and the `error rpc error: code = Unknown desc = unexpected HTTP status code received from server: 0 (); malformed header: missing HTTP content-type` returned by the grpc client. This error is very frequent and seriously affects the use of our production environment.  The host is: ep-mzinkvxm.endpoint.windmill.com here is my route config: ``` - name: grpc priority: 1 match: hosts: - {{ .Release.Name }}.endpoint.windmill.com paths: - "/*" backends: - serviceName: {{ .Release.Name }} servicePort: grpc ``` here is the config i copy from apisix-admin: ``` { "uris": ["/*"], "name": "default_ep-mzinkvxm_grpc", "desc": "Created by apisix-ingress-controller, DO NOT modify it manually", "priority": 1, "hosts": ["[ep-mzinkvxm.endpoint.windmill.com](http://ep-mzinkvxm.endpoint.windmill.com/)"], "upstream_id": "c54210dc", "labels": { "managed-by": "apisix-ingress-controller" }, "status": 1 } ``` and there is some error I found from apisix-ingress-controller, i don't know if this error is related to this phenomenon. ``` 2023-03-29T20:35:08+08:00 warn ingress/apisix_route.go:440 sync ApisixRoute failed, will retry {"object": {"Type":1,"Object":{"Key":"default/ep-fuphptti","OldObject":null,"GroupVersion":"[apisix.apache.org/v2](http://apisix.apache.org/v2)"},"OldObject":null,"Tombstone":null}, "error": "6 errors occurred:\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [cf3c7922]: has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [b83b49b4]: has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 40 0; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [2132180e]: has no healthy etcd endpoint available\"}\n\n\n"} ``` ### Expected Behavior I hope this mistake won't happen again. ### Error Logs _No response_ ### Steps to Reproduce 1. Running APIsix on k8s using Helm Chart. 2. Configure APIsix routes using YAML ``` {{- if .Values.ingress.enabled -}} apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: {{ .Release.Name }} spec: http: - name: http priority: 3 match: paths: - /{{ .Release.Name }}/http/* backends: - serviceName: {{ .Release.Name }} servicePort: http plugins: - name: proxy-rewrite enable: true config: regex_uri: ["^/{{ .Release.Name }}/http/(.*)", "/$1"] - name: api-breaker enable: true config: break_response_code: 502 unhealthy: http_statuses: [500, 503] failure: 3 healthy: http_statuses: [200] successes: 1 - name: grpc priority: 1 match: hosts: - {{ .Release.Name }}.endpoint.windmill.com paths: - "/*" backends: - serviceName: {{ .Release.Name }} servicePort: grpc - name: metrics priority: 2 match: paths: - /{{ .Release.Name }}/metrics/* backends: - serviceName: {{ .Release.Name }} servicePort: metrics plugins: - name: proxy-rewrite enable: true config: regex_uri: ["^/{{ .Release.Name }}/metrics/(.*)", "/$1"] --- apiVersion: apisix.apache.org/v2 kind: ApisixUpstream metadata: name: {{ .Release.Name }} spec: loadbalancer: type: ewma retries: {{ .Values.ingress.retries }} timeout: connect: {{ .Values.ingress.timeout.connect }} send: {{ .Values.ingress.timeout.send }} read: {{ .Values.ingress.timeout.read }} portLevelSettings: - port: 8000 scheme: http - port: 8001 scheme: grpc - port: 8002 scheme: http {{- end }} ``` 3. use grpc client to request <img width="930" alt="image" src="https://user-images.githubusercontent.com/41530068/229771042-b9821be6-38aa-4445-a3ca-1140adc178cc.png";> ### Environment - APISIX version (run `apisix version`): <img width="793" alt="image" src="https://user-images.githubusercontent.com/41530068/229771293-341789c2-43ea-40bb-94c3-87cd7b665289.png";> - Operating system (run `uname -a`): `Linux apisix-656c7b655c-dlqgk 4.14.0_1-0-0-50 #1 SMP Wed Jun 1 18:07:32 CST 2022 x86_64 Linux` - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): <img width="1919" alt="image" src="https://user-images.githubusercontent.com/41530068/229771376-d95d9e25-37ff-4e7c-83cf-bd6ebb8537d3.png";> - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):`3.5.4-debian-11-r14` - APISIX Dashboard version, if relevant: `apisix-dashboard:2.13-alpine` - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): `/usr/local/bin/luarocks 3.8.0` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/41530068/229771042-b9821be6-38aa-4445-a3ca-1140adc178cc.png"](https://user-images.githubusercontent.com/41530068/229771042-b9821be6-38aa-4445-a3ca-1140adc178cc.png"){kind=link}
![https://user-images.githubusercontent.com/41530068/229771293-341789c2-43ea-40bb-94c3-87cd7b665289.png"](https://user-images.githubusercontent.com/41530068/229771293-341789c2-43ea-40bb-94c3-87cd7b665289.png"){kind=link}
![https://user-images.githubusercontent.com/41530068/229771376-d95d9e25-37ff-4e7c-83cf-bd6ebb8537d3.png"](https://user-images.githubusercontent.com/41530068/229771376-d95d9e25-37ff-4e7c-83cf-bd6ebb8537d3.png"){kind=link}