Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
Closed #2632. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#event-4495885186___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
Closing this one, commit was pushed to stable branches. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804868440___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
I switched to using http_client module and have not had any crashes since February, so it that module is affected, it's in a different way. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804438635___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
@miconda I don't think it is needed to modify any of the flags except the one controlling multiplexing. @grumvalski I had a quick look at the http_client module, and don't think it is affected by this issue. It uses curl_easy_perform(), which is a blocking call. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804270243___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
If there is agreement I'll merge this one and backport it to stable branches. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804259979___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
@miconda: not sure that http_client is affected too, being the problem in http_async_client the fact tha a socket is bound to a specific easy handler and an event is attached to it via libevent. But I think @oej is better qualified to answer :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804240779___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
@grumvalski, @bjokax: just wanted to ensure flags listed around here are evaluated to see if worth setting them, not necessary for fixing the issue, but from the perspective of what Kamailio needs/expects from this function/module. I am fine to just go with what is considered better, then probably push it to the http_client module. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804121697___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
@grumvalski I think the fix in your branch is good enough! It is the multiplexing that is the problem, not http2. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804116204___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
@miconda: in this case we should also disable NPN negotiation: https://curl.se/libcurl/c/CURLOPT_SSL_ENABLE_NPN.html, or even disabling HTTP/2 completely setting CURLOPT_HTTP_VERSION (current curl default value is CURLOPT_HTTP_VERSION_2TLS). I didn't want to be so extreme :) @bjokax, thanks for testing it. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804101356___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
We have now tested https://github.com/kamailio/kamailio/tree/grumvalski/disable_http2_multiplexing and it seems to solve our problems! @miconda I would ague that this is a better solution than disabling TLS ALPN. Now we can continue use http2, which is a more efficient protocol than http 1.1, but just disable the multiplexing, until we have the full refactoring in place. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-804016645___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
@grumvalski: have you checked if also makes sense to set the flag suggested by @bjokax CURLOPT_SSL_ENABLE_ALPN ? * https://curl.se/libcurl/c/CURLOPT_SSL_ENABLE_ALPN.html Wondering if there is any benefit of allowing protocol negotiation in Kamailio case. Moreover, maybe we have to set it to http_client module - @oej: any opinion on http_client module? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-803980441___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
Hi, althought I have not been able to reproduce the crash, I think that the problem, as highlighted by @bjokax, is the not correct handling of HTTP/2 multiplexing. The module was written before HTTP/2 was out so it has never been considered. Support for it would require some important refactoring of the module, I'll start seeing if I can work on it for the next major release. Meanwhile we can disable multiplexing at all (this can also be backported to current stable branches while a refactor to support multiplexing wouldn't). I've pushed such fix in this branch https://github.com/kamailio/kamailio/tree/grumvalski/disable_http2_multiplexing. @amessina @bjokax : can you try it? Since I've not been able for now to reproduce the crash I don't know if it solves yours. Thank you. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-803621237___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
I nicer solution would be to fully implement support for http2! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-803385884___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
Thank you @bjokax, you are correct -- I was not aware of the implementation detail related to http2, though it's promising that your `curl_easy_setopt(cell->easy, CURLOPT_SSL_ENABLE_ALPN, 0L);` may resolve the issue. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-803143438___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
I can confirm that patching http_async_client module by adding `curl_easy_setopt(cell->easy, CURLOPT_SSL_ENABLE_ALPN, 0L);` solved our problems. Now we don't see any spinning Kamailio processes anymore. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-803055188___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
Thank you for the report. I'll try to have a look on it in the next couple of days. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-802922909___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
We suspect that the issue only happens with http2. @amessina I can see that the server you connect to is also using http2. Looking at the implementation of http_async_client it doesn't seem to handle http2 properly. If you understood the implementation correctly these is an association between the Curl_easy and the fd, when registering in libevents. That does not work in http2, where several requests are multiplexed over the same TLS connection. A quick fix could be to disable ALPN `curl_easy_setopt(curl_, CURLOPT_SSL_ENABLE_ALPN, 0L);`. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-802806829___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
We see the same issue on both 5.3 and 5.4: ``` version: kamailio 5.3.8 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 8.3.0 ``` ``` version: kamailio 5.4.4 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 8.3.0 ``` ` Linux sbc-stateless-ig-wn7p 4.19.0-14-cloud-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-802093018___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
I've have similar crashes on syslog(). But I don't use HTTP(S).
### Troubleshooting
Debugging Data
```
(lldb) target create "kamailio" --core "/var/coredump/986/kamailio.4087.core"
Core file '/var/coredump/986/kamailio.4087.core' (x86_64) was loaded.
(lldb) bt all
* thread #1, name = 'kamailio', stop reason = signal SIGSEGV
* frame #0: 0x0008009fa56d libc.so.7`__je_tcache_bin_flush_small
[inlined] extent_arena_get(extent=0x) at extent_inlines.h:39:43
frame #1: 0x0008009fa56d
libc.so.7`__je_tcache_bin_flush_small(tsd=, tcache=,
tbin=0x000800d1a2a8, binind=, rem=33) at jemalloc_tcache.c:123
frame #2: 0x0008009fa17c
libc.so.7`__je_tcache_event_hard(tsd=, tcache=0x000800d1a250)
at jemalloc_tcache.c:54:4
frame #3: 0x000800a37934 libc.so.7`__malloc [inlined]
arena_malloc(tsdn=0x000800d1a090, arena=0x,
size=, ind=, zero=false, tcache=,
slow_path=false) at arena_inlines_b.h:0
frame #4: 0x000800a378a7 libc.so.7`__malloc [inlined]
iallocztm(tsdn=0x000800d1a090, size=, ind=,
zero=false, tcache=, is_internal=false, arena=0x,
slow_path=false) at jemalloc_internal_inlines_c.h:53
frame #5: 0x000800a378a7 libc.so.7`__malloc [inlined]
imalloc_no_sample(sopts=, dopts=, tsd=,
size=, usize=, ind=) at
jemalloc_jemalloc.c:1713
frame #6: 0x000800a378a7 libc.so.7`__malloc [inlined]
imalloc_body(sopts=, dopts=, tsd=) at
jemalloc_jemalloc.c:1909
frame #7: 0x000800a378a7 libc.so.7`__malloc [inlined]
imalloc(sopts=, dopts=) at jemalloc_jemalloc.c:2009
frame #8: 0x000800a3768b libc.so.7`__malloc(size=) at
jemalloc_jemalloc.c:2042
frame #9: 0x000800acaf34 libc.so.7`__smakebuf(fp=0x000800d06010) at
makebuf.c:73:11
frame #10: 0x000800acae39 libc.so.7`__swsetup(fp=0x000800d06010) at
wsetup.c:82:3
frame #11: 0x000800a67b53 libc.so.7`__vfprintf(fp=0x000800d06010,
locale=0x000800ade698, fmt0="", ap=0x7fff8cf0) at vfprintf.c:462:6
frame #12: 0x000800a678c5 libc.so.7`vfprintf_l(fp=0x000800d06010,
locale=0x000800ade698, fmt0="", ap=0x7fff8cf0) at vfprintf.c:285:9
frame #13: 0x000800a6f113 libc.so.7`fprintf(fp=,
fmt=) at fprintf.c:57:8
frame #14: 0x000800aa8f01 libc.so.7`vsyslog [inlined] vsyslog1(pri=135,
fmt="", ap=0x7fff9c50) at syslog.c:173:8
frame #15: 0x000800aa8e4c libc.so.7`vsyslog(pri=, fmt="",
ap=0x7fff9c50) at syslog.c:363
frame #16: 0x000800aa8dcd libc.so.7`syslog(pri=,
fmt=) at syslog.c:129:2
frame #17: 0x006655dc
kamailio`handle_tcp_child(tcp_c=0x0008014dcf08, fd_i=-1) at
tcp_main.c:3559:2
frame #18: 0x0065b7e9 kamailio`handle_io(fm=0x0008014e6080,
ev=1, idx=-1) at tcp_main.c:4523:8
frame #19: 0x00644502
kamailio`io_wait_loop_kqueue(h=0x008dbf50, t=5, repeat=0) at
io_wait.h:1187:9
frame #20: 0x0063cb62 kamailio`tcp_main_loop at tcp_main.c:4824:5
frame #21: 0x002e973a kamailio`main_loop at main.c:1779:5
frame #22: 0x002f7650 kamailio`main(argc=7,
argv=0x7fffec70) at main.c:2856:6
frame #23: 0x002d1af0 kamailio`_start(ap=,
cleanup=) at crt1.c:76:7
```
Log Messages
```
03:20:21.349512 4087 - - DEBUG: [core/tcp_main.c:4130]: send2child():
selected tcp worker idx:1 proc:26 pid:4080 for activity on
[tcp:91.217.xx.yy:5060], 0x802e70718
03:20:27.364691 4087 - - DEBUG: [core/tcp_main.c:3560]:
handle_tcp_child(): reader response= 802e70718, 1 from 1
03:20:27.364727 4087 - - DEBUG: [core/io_wait.h:375]: io_watch_add():
DBG: io_watch_add(0x8dbf50, 53, 2, 0x802e70718), fd_no=43
03:20:27.364746 4087 - - DEBUG: [core/tcp_main.c:3687]:
handle_tcp_child(): CONN_RELEASE 0x802e70718 refcnt= 1
03:20:29.740878 4062 - - DEBUG: {1 1001 SUBSCRIBE
ceccb6ad-add35f9c@10.10.10.251} [core/socket_info.c:646]:
grep_sock_info(): checking if host==us: 13==13 && [78.37.aa.bb] ==
[91.217.xx.yy]
03:20:51.496077 4087 - - DEBUG: [core/io_wait.h:600]: io_watch_del():
DBG: io_watch_del (0x8dbf50, 53, -1, 0x0) fd_no=44 called
03:20:51.496127 4087 - - DEBUG: [core/tcp_main.c:4457]:
handle_tcpconn_ev(): sending to child, events 1
03:20:51.496147 4087 - - DEBUG: [core/tcp_main.c:4130]: send2child():
selected tcp worker idx:2 proc:27 pid:4081 for activity on
[tcp:91.217.xx.yy:5060], 0x802e70718
03:20:58.190139 4054 - - ALERT: [main.c:777]: handle_sigs(): child
process 4087 exited by a signal 11
```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
kamailio 5.4.3 (x86_64/freebsd) e19ae3
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144,
Re: [sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
Another using http_async_client -- seems like much of this is related to libevent [gdb.203123.txt](https://github.com/kamailio/kamailio/files/5983969/gdb.203123.txt) I've switched over to http_client for now. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2632#issuecomment-779435087___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
[sr-dev] [kamailio/kamailio] Core dumps possibly related to #2616 or http_async_query (#2632)
### Description
When working atop the 5.4 branch patched with `--no-atexit` option from
https://github.com/kamailio/kamailio/issues/2616#issuecomment-771811762
applied, I've been testing the use of http_async_query to pass out MESSAGE
requests to selected destinations to an API via HTTPS. I experienced the
following core dumps during the time where a higher number (may 10 or so) of
these async queries may have been happening around the same time. I am not
sure however how to interpret these and they may have had nothing to do with
#2616.
Of note for process `23694`, the logs spewed `INFO: http_async_client
[http_multi.c:87]: event_cb(): Cell for handler 0x7f834b7eda08 not found in
table` thousands of times, spiking CPU until Kamailio finally quit with sig 11.
Debugging Data
For core dump of process `2193`:
```
#0 0x7f367f64ea23 in free_cell_helper (dead_cell=0x7f3683d34b48, silent=0,
fname=0x7f367f770902 "timer.c", fline=643) at h_table.c:186
b = 0x7f367f64c26d
"\211E\374\203}\374\002\017\224\300\017\266\300H\205\300t6H\213E\350H\203\354\bj"
i = 1
rpl = 0x7ffd51f7fd40
tt = 0x7ffd51f7fd60
foo = 0x0
cbs = 0x0
cbs_tmp = 0x7f3680154a20
__func__ = "free_cell_helper"
#1 0x7f367f71b738 in wait_handler (ti=2051374262, wait_tl=0x7f3683d34bd0,
data=0x7f3683d34b48) at timer.c:643
p_cell = 0x7f3683d34b48
ret = 2143519712
unlinked = 0
rcount = 1
__func__ = "wait_handler"
#2 0x006ae985 in timer_list_expire (t=2051374262, h=0x7f367fc7c4c0,
slow_l=0x7f367fc7e398, slow_mark=16842) at core/timer.c:857
tl = 0x7f3683d34bd0
ret = 32566
#3 0x006aee48 in timer_handler () at core/timer.c:922
saved_ticks = 2051374262
run_slow_timer = 0
i = 458
__func__ = "timer_handler"
#4 0x006af30a in timer_main () at core/timer.c:961
No locals.
#5 0x0042c59c in main_loop () at main.c:1769
i = 2
pid = 0
si = 0x0
si_desc = "udp receiver child=1
sock=[2603:300a:134:50e0::3]:5060\000\000\300\261ʈ6\177\000\000\360\005G\210\066\177\000\000Z\262ʈ6\177\000\000\060\000\000\000\060\000\000\000\b\001\370Q\375\177\000\000
\000\370Q\375\177\000\000\000\035\064\371\364\245\242.\350\301\201\000\000\000\000\000\270\251\206\210\066\177\000"
nrprocs = 2
woneinit = 1
__func__ = "main_loop"
#6 0x00435d22 in main (argc=8, argv=0x7ffd51f80648) at main.c:2876
cfg_stream = 0x19e5120
c = -1
r = 0
tmp = 0x7ffd51f81ee5 ""
tmp_len = 896
port = 896
proto = 896
ahost = 0x0
aport = 0
options = 0x7f3de8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 504719050
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 2
n_lst = 0x0
p = 0xc2
st = {st_dev = 26, st_ino = 1205, st_nlink = 2, st_mode = 16872, st_uid
= 985, st_gid = 983, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096,
st_blocks = 0, st_atim = {tv_sec = 1612794079, tv_nsec = 14695130}, st_mtim =
{tv_sec = 1612794079, tv_nsec = 14695130}, st_ctim = {tv_sec = 1612794079,
tv_nsec = 14695130}, __glibc_reserved = {0, 0, 0}}
tbuf =
"@\002\370Q\375\177\000\000\000\000\000\000\000\000\000\000@\002\370Q\375\177",
'\000' ,
"\260\067\361\210\066\177\000\000\350\357\363\210\066\177\000\000\b\005\364\210\066\177\000\000\340\071\361\210\066\177\000\000\025\257\362\210\066\177\000\000$V\273\210\066\177\000\000\354k\363\210\066\177\000\000\336k\363\210\066\177\000\000\205\317c\t\000\000\000\000\300C\360\210\066\177\000\000߯\362\210\066\177\000\000\000\000\000\000\254\202\226\006\334@\273\210\066\177\000\000\000\000\000\000\000\000\000\000\300C\360\210\066\177\000\000\001\000\000\000\000\000\000\000\005\215.\201\267e\325\000\240\001\364\210\066\177\000\000\370\377\377\377\377\377\377\377\240\001\364\210\066\177\000\000R"...
option_index = 11
long_options = {{name = 0x7f62a6 "help", has_arg = 0, flag = 0x0, val =
104}, {name = 0x7f1523 "version", has_arg = 0, flag = 0x0, val = 118}, {name =
0x7f62ab "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7f62b1
"subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7f62b7 "substdef",
has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7f62c0 "substdefs", has_arg =
1, flag = 0x0, val = 1027}, {
name = 0x7f62ca "server-id", has_arg = 1, flag = 0x0, val = 1028},
{name = 0x7f62d4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name =
0x7f62df "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7f62e8
"log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7f62f3 "debug",
has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7f62f9 "no-atexit", has_arg =
0, flag = 0x0, val = 1033}, {
name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
