math-chao opened a new issue, #9464:
URL: https://github.com/apache/apisix/issues/9464
### Description
version: apisix:2.15.0-centos in k8s
run for 13 days, and Program deadlock,
error log
prometheus.lua:920: log_error(): Error while setting
'node_info{hostname=""}' to '1': 'no memory', client: xxx.xxxx.xxxx, server: ,
request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "xxxx"
prometheus.lua:920: log_error(): Error while setting
'etcd_modify_indexes{key="x_etcd_index"}' to '8553921': 'no memory', client:
xxx.xxxx.xxx, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1",
host: "xxxx"
debug by strace, found a lot of futex calls
futex(0x7fc9d4ded080, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily
unavailable)
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily
unavailable)
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
debug by perf
Samples: 31K of event 'cycles', Event count (approx.): 18419237386
Children Self Command Shared Object Symbol
+ 87.95% 0.00% openresty [unknown] [k]
0x0000000000000080
+ 84.44% 83.62% openresty nginx [.] ngx_shmtx_lock
+ 4.85% 0.10% openresty [kernel.kallsyms] [k]
entry_SYSCALL_64_after_hwframe
+ 4.57% 0.03% openresty [kernel.kallsyms] [k] do_syscall_64
+ 4.36% 0.07% openresty [kernel.kallsyms] [k] __x64_sys_futex
+ 4.30% 0.08% openresty [kernel.kallsyms] [k] do_futex
pstack results are as follows
#0 0x0000000000449bf3 in ngx_shmtx_lock (mtx=0x7fc9d4ded068) at
src/core/ngx_shmtx.c:78
#1 0x0000000000572caf in ngx_meta_lua_ffi_shdict_get (zone=0x1281138,
key=0x7fc9d1d69080 "__ngx_prom__delete_count", key_len=24,
value_type=0x7fc9de85f828, str_value_buf=0x7fc9de844380,
str_value_len=0x7fc9de875ef8, num_value=0x7fc9de85f868,
user_flags=0x7fc9de85f848, get_stale=0, is_stale=0x7fc9de8636f8,
err=0x7fc9de8723a8) at
/tmp/tmp.YN7822nAJt/openresty-1.21.4.1/../apisix-nginx-module-1.9.0/src/meta/ngx_meta_lua_shdict.c:1631
#2 0x00007fca19caa8d7 in ?? ()
#3 0x00007fc9de85f868 in ?? ()
#4 0x00007fc9de85f848 in ?? ()
#5 0x0000000000000000 in ?? ()
What's the problem？help please
### Environment
- APISIX version (run `apisix version`): 2.15.0-centos
- Operating system (run `uname -a`): Linux 5.10.131-1.el8.x86_64 #1 SMP Sun
Jul 17 11:20:32 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
