huayi550 opened a new issue, #7349: URL: https://github.com/apache/apisix/issues/7349
### Current Behavior On the grafanal panel，we found error for nginx metrics such as below  so we check the prometheus target of one APISIX instance `http://10.xx.xx.xx:9091/apisix/prometheus/metrics` ``` # HELP apisix_nginx_http_current_connections Number of HTTP connections # TYPE apisix_nginx_http_current_connections gauge apisix_nginx_http_current_connections{state="accepted"} 5011060 apisix_nginx_http_current_connections{state="active"} 6 apisix_nginx_http_current_connections{state="handled"} 5011060 apisix_nginx_http_current_connections{state="reading"} 0 apisix_nginx_http_current_connections{state="waiting"} 1 apisix_nginx_http_current_connections{state="writing"} 5 # HELP apisix_nginx_metric_errors_total Number of nginx-lua-prometheus errors # TYPE apisix_nginx_metric_errors_total counter apisix_nginx_metric_errors_total 1372 # HELP apisix_node_info Info of APISIX node # TYPE apisix_node_info gauge apisix_node_info{hostname="ecs-apisix-0001"} 1 ``` - Usage on production environment | APISIX Instance | Etcd Cluster Instance | Upstream | Routes| |-- | -- | -- | --| | 3 | 3 | 28 | 614| <!--EndFragment--> </body> </html> Then we check the error on /apisix/logs/error.log ，found the logs about prometheus below： ### Expected Behavior _No response_ ### Error Logs ``` 2022/06/29 10:51:16 [error] 32404#32404: *92610688 [lua] prometheus.lua:872: log_error(): Unexpected error adding a key: no memory while logging request, client: 10.11.0.43, server: _, request: "POST /parkMap/modifyMapName HTTP/1.0", upstream: "http://96.11.0.4:8889/parkMap/modifyMapName";, host: "apigw.*****.com.cn" 2022/06/29 10:51:16 [error] 32404#32404: *92610688 [lua] prometheus.lua:872: log_error(): Unexpected error adding a key: no memory while logging request, client: 10.11.0.43, server: _, request: "POST /parkMap/modifyMapName HTTP/1.0", upstream: "http://96.11.0.4:8889/parkMap/modifyMapName";, host: "apigw.*****.com.cn" 2022/06/29 10:51:16 [error] 32404#32404: *92610688 [lua] prometheus.lua:872: log_error(): Unexpected error adding a key: no memory while logging request, client: 10.11.0.43, server: _, request: "POST /parkMap/modifyMapName HTTP/1.0", upstream: "http://96.11.0.4:8889/parkMap/modifyMapName";, host: "apigw.*****.com.cn" ``` ### Steps to Reproduce we have enabled the global plugin : `prometheus` ，the config is below ``` { "disable": false, "prefer_name": false } ``` ### Environment - APISIX version (run `apisix version`): 2.13.1 - Operating system (run `uname -a`): CentOS7.9 - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): openresty/1.19.9.1 - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): 3.5.2 - APISIX Dashboard version, if relevant: 2.13.0 - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
