Adrianckwai opened a new issue, #12282: URL: https://github.com/apache/apisix/issues/12282
### Description Hi Apisix team, We have encountered an issue, where our service A request to service B via APISIX using GRPC scheme. We noticed that our APISIX will delayed or sometimes unable to receive request, before upstream to our service B. This issue happened intermittently and has been impacting our production environment. Hope we can get help on this. Thanks! **Trace for delayed APISIX request:**  **Trace for DEADLINE_EXCEEDED request:** <img width="1039" alt="Image" src="https://github.com/user-attachments/assets/fd003002-7160-4677-bca6-4b676bfd70b9"; /> **Error msg from service A:** `{"level":50,"time":1748229084607,"instance_id":"1","trace_id":"885e97e4c157355063e62855bd85483e","span_id":"a9942b384a76f518","trace_flags":1,"msg"/:"Error in grpcClientCall: 4 DEADLINE_EXCEEDED: Deadline exceeded after 5.000s,name resolution: 0.001s,metadata filters: 0.001s,LB pick: 0.004s,remote_addr=10.253.24.104:80 Error"}` **APISIX route config:** $ k get ar -n fpms-nt auth-apiserver-internal-route -o yaml ```shell apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: annotations: argocd.argoproj.io/tracking-id: fpms-nt:apisix.apache.org/ApisixRoute:fpms-nt/auth-apiserver-internal-route creationTimestamp: "2025-04-17T06:52:15Z" generation: 24 name: auth-apiserver-internal-route namespace: fpms-nt resourceVersion: "141266233" uid: 51c15c9d-0d7f-42cd-9062-d12c1718696f spec: http: - backends: - serviceName: auth-apiserver-cs servicePort: 80 weight: 0 - serviceName: auth-apiserver-ss servicePort: 80 weight: 100 match: hosts: - fpms-nt-internal.fpms100.me - fpms-nt-internal.fpms10.me paths: - /auth.* name: auth-apiserver-internal-route plugins: - enable: true name: prometheus - config: sampler: name: always_on options: fraction: 0.1 root: name: always_on options: fraction: 0.1 enable: true name: opentelemetry - config: count: 5000 key: http_service key_type: var policy: redis redis_host: prod-fpms-nt-apisix-valkey-sg.7odqjm.ng.0001.apse1.cache.amazonaws.com redis_port: 6379 rejected_code: 429 time_window: 1 enable: true name: limit-count status: conditions: - message: Sync Successfully observedGeneration: 18 reason: ResourcesSynced status: "True" type: ResourcesAvailable ``` **APISIX upstream config:** $ k get apisixupstream -n fpms-nt auth-apiserver-ss -o yaml ```shell apiVersion: apisix.apache.org/v2 kind: ApisixUpstream metadata: annotations: argocd.argoproj.io/tracking-id: fpms-nt:apisix.apache.org/ApisixUpstream:fpms-nt/auth-apiserver-ss creationTimestamp: "2025-04-17T06:52:15Z" generation: 1 name: auth-apiserver-ss namespace: fpms-nt resourceVersion: "142996228" uid: d72751e7-8cb9-4dc7-92db-79cc42cc6aa4 spec: loadbalancer: type: least_conn scheme: grpc status: conditions: - message: not found observedGeneration: 1 reason: ResourceSyncAborted status: "False" type: ResourcesAvailable ``` ### Environment - APISIX version (run `apisix version`): 3.12.0 - Operating system (run `uname -a`): Linux apisix-588d5777db-t6ncn 6.1.134-152.225.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 7 09:10:59 UTC 2025 x86_64 GNU/Linux - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): nginx version: openresty/[1.27.1.1](http://1.27.1.1/) - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): {"etcdserver":"3.5.10","etcdcluster":"3.5.0"} - APISIX Dashboard version, if relevant: - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
