Hi Bogdan,
good hit on the apport service - thanks!
here's the trace we get from one such crash:
[New LWP 1130]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f
/etc/opensips/opensips.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()
(gdb) bt full
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#1 0x00007f334e06aad5 in acc_aaa_cdrs (dlg=<optimized out>, msg=<optimized
out>, ctx=0x7f334fcc52c8)
at /build/opensips-3.4.2/modules/acc/acc.c:965
i = <optimized out>
j = <optimized out>
ret = 6
res = -1
nr_leg_vals = <optimized out>
start_time = {tv_sec = 1702044564, tv_usec = 417047}
core_s = {s = 0x7f33705e9878 "\006", len = 101}
leg_s = {s = 0x0, len = <optimized out>}
extra_s = {s = 0x0, len = <optimized out>}
send = 0x0
offset = <optimized out>
av_type = 32563
r_stat = <optimized out>
locked = 0
duration = 1338791072
ms_duration = 0
extra = <optimized out>
error = <optimized out>
__FUNCTION__ = "acc_aaa_cdrs"
#2 0x00007f334e078ade in acc_cdr_cb (t=0x0, type=5, ps=0x7ffdb12e09e0)
at /build/opensips-3.4.2/modules/acc/acc_logic.c:986
ctx = 0x7f334fcc52c8
dlg = 0x7f334fcc54e0
__FUNCTION__ = "acc_cdr_cb"
#3 0x00007f334f8142a0 in run_any_trans_callbacks
(list=list@entry=0x7f334fcc6d58, type=type@entry=128,
trans=trans@entry=0x7f334fcc6ce8, req=0x7f334fcc8770,
rpl=rpl@entry=0x7f33705e7498, code=code@entry=200)
at /build/opensips-3.4.2/modules/tm/t_hooks.c:214
params = {req = 0x7f334fcc8770, rpl = 0x7f33705e7498, code = 200, param
= 0x7f334fcc5270,
extra1 = 0x7ffdb12e0b40, extra2 = 0x7f334fcc6e00}
cbp = 0x7f334fcc5260
backup = 0x55c8f7a02b00 <global_avps.lto_priv>
trans_backup = 0x7f334fcc6ce8
__FUNCTION__ = "run_any_trans_callbacks"
#4 0x00007f334f818443 in run_trans_callbacks (type=type@entry=128,
trans=trans@entry=0x7f334fcc6ce8, req=<optimized out>,
rpl=rpl@entry=0x7f33705e7498, code=code@entry=200) at
/build/opensips-3.4.2/modules/tm/t_hooks.c:233
No locals.
#5 0x00007f334f818581 in run_trans_callbacks_locked (code=200, rpl=0x7f33705e7498,
req=<optimized out>, trans=0x7f334fcc6ce8, type=128) at
/build/opensips-3.4.2/modules/tm/t_hooks.c:255
No locals.
#6 run_trans_callbacks_locked (type=128, trans=0x7f334fcc6ce8, req=<optimized
out>, rpl=0x7f33705e7498, code=200) at
/build/opensips-3.4.2/modules/tm/t_hooks.c:246
No locals.
#7 0x00007f334f82ef22 in relay_reply (t=0x7f334fcc6ce8, p_msg=<optimized out>,
branch=<optimized out>, msg_status=<optimized out>, cancel_bitmap=<optimized out>) at
/build/opensips-3.4.2/modules/tm/t_reply.c:1408
relay = 0
save_clone = 0
buf = 0x7f33705e9620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP
212.13.249.94;received=212.13.249.94;rport=5060;branch=z9hG4bKUtNtK5Nj897rr\r\nFrom:
<sip:386980731395"...
res_len = 536
relayed_code = 200
relayed_msg = 0x7f33705e7498
bm = {to_tag_val = {s = 0x7f336f891af8 "\240\350\240\367\310U", len =
0}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f334fcc6de8
cb_s = {s = 0x7f33705e9620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP
212.13.249.94;received=212.13.249.94;rport=5060;branch=z9hG4bKUtNtK5Nj897rr\r\nFrom:
<sip:386980731395"..., len = 536}
text = {s = 0x0, len = -140509096}
__FUNCTION__ = "relay_reply"
#8 0x00007f334f830835 in reply_received (p_msg=0x7f33705e7498) at
/build/opensips-3.4.2/modules/tm/t_reply.c:1681
msg_status = <optimized out>
last_uac_status = 0
branch = 0
reply_status = <optimized out>
timer = 0
cancel_bitmap = 0
uac = 0x7f334fcc6ec0
t = 0x7f334fcc6ce8
backup_list = <optimized out>
has_reply_route = <optimized out>
old_route_type = <optimized out>
__FUNCTION__ = "reply_received"
#9 0x000055c8f770e0e5 in forward_reply (msg=0x7f33705e7498) at
/build/opensips-3.4.2/forward.c:499
new_buf = <optimized out>
to = <optimized out>
new_len = 1
mod = 0x7f336f891af8
proto = <optimized out>
id = <optimized out>
send_sock = <optimized out>
s = <optimized out>
len = <optimized out>
error = <optimized out>
skip = <optimized out>
error0 = <optimized out>
__FUNCTION__ = "forward_reply"
#10 0x000055c8f775542d in receive_msg (buf=<optimized out>, len=<optimized out>,
rcv_info=<optimized out>, existing_context=0x0, msg_flags=<optimized out>) at
/build/opensips-3.4.2/receive.c:278
msg = 0x7f33705e7498
start = {tv_sec = 139858358410128, tv_usec = 139859026441837}
rc = 3
old_route_type = <optimized out>
tmp = <optimized out>
in_buff = {s = 0x55c8f7b24320 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
213.253.120.65:5060;branch=z9hG4bK4e83.b2fe4a51.0\r\nVia: SIP/2.0/UDP
212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP
212.13.249.94;received=212.13.24"..., len = 612}
ctx = 0x7f33705e8bc8
__FUNCTION__ = "receive_msg"
#11 0x000055c8f78addd7 in udp_read_req (si=<optimized out>, bytes_read=<optimized
out>) at net/proto_udp/proto_udp.c:186
ri = {src_ip = {af = 2, len = 4, u = {addrl = {94317692413908, 94321635752912}, addr32 = {210593748, 21960, 4153932752, 21960},
addr16 = {26580, 3213, 21960, 0, 64464, 63383, 21960, 0}, addr = "\324g\215\f\310U\000\000\320\373\227\367\310U\000"}},
dst_ip = {af = 2, len = 4, u = {addrl = {1098448341, 0}, addr32 = {1098448341, 0, 0, 0}, addr16 = {64981, 16760, 0, 0, 0, 0, 0, 0},
addr = "\325\375xA", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto = 1, proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\324g\215\f\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 210593748}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 210593748, sin6_addr = {__in6_u = {__u6_addr8 =
"\000\000\000\000\000\000\000\000 +\240\367\310U\000", __u6_addr16 = {0, 0, 0, 0, 11040, 63392, 21960, 0}, __u6_addr32 = {0,
0, 4154469152, 21960}}}, sin6_scope_id = 2973633109}}, bind_address = 0x7f336f8b52d8}
len = <optimized out>
tmp = <optimized out>
fromlen = 16
p = <optimized out>
msg = {s = 0x55c8f7b24320 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
213.253.120.65:5060;branch=z9hG4bK4e83.b2fe4a51.0\r\nVia: SIP/2.0/UDP
212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP
212.13.249.94;received=212.13.24"..., len = 612}
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
213.253.120.65:5060;branch=z9hG4bK4e83.b2fe4a51.0\r\nVia: SIP/2.0/UDP
212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP
212.13.249.94;received=212.13.24"...
__FUNCTION__ = "udp_read_req"
#12 0x000055c8f78c75da in handle_io.constprop.1 (fm=0x7f336f8e7788,
event_type=event_type@entry=1, idx=<optimized out>) at net/net_udp.c:295
n = 0
read = 0
__FUNCTION__ = <optimized out>
#13 0x000055c8f78d4c40 in io_wait_loop_epoll.constprop.1.isra.0 (repeat=<optimized out>,
t=<optimized out>, h=<optimized out>) at net/../io_wait_loop.h:305
ret = <optimized out>
n = <optimized out>
r = 0
i = <optimized out>
e = <optimized out>
ep_event = {events = 2, data = {ptr = 0xf7a0320000000000, fd = 0, u32 =
0, u64 = 17843316699223293952}}
fd = <optimized out>
curr_time = 78
error = <optimized out>
__FUNCTION__ = <optimized out>
#14 0x000055c8f78a42b5 in udp_start_processes (chd_rank=<optimized out>,
startup_done=<optimized out>) at net/net_udp.c:528
si = <optimized out>
p_id = <optimized out>
i = <optimized out>
p = <optimized out>
ifp_udp_rcv = {proc_desc = 0x55c8f796c046 "UDP receiver", flags = 4,
type = TYPE_UDP}
error = <optimized out>
__FUNCTION__ = "udp_start_processes"
#15 0x000055c8f76d575d in main_loop () at /build/opensips-3.4.2/main.c:239
startup_done = 0x0
last_check = 0
rc = <optimized out>
chd_rank = <optimized out>
startup_done = <optimized out>
last_check = <optimized out>
rc = <optimized out>
profiling_handler = <optimized out>
__FUNCTION__ = <optimized out>
error = <optimized out>
#16 main (argc=<optimized out>, argv=<optimized out>) at
/build/opensips-3.4.2/main.c:968
c = <optimized out>
r = <optimized out>
tmp = 0x7ffdb12e2e53 ""
tmp_len = <optimized out>
port = <optimized out>
proto = 0
protos_no = <optimized out>
options = 0x55c8f78ea290
"f:cCm:M:b:l:n:N:rRvdDFEVhw:t:u:g:p:P:G:W:o:a:k:s:"
ret = <optimized out>
seed = 3254223836
rfd = <optimized out>
procs_no = <optimized out>
__FUNCTION__ = "main"
On 08.12.23 15:16, Bogdan-Andrei Iancu wrote:
Simon, I do not need the corefile, usually it is huge - as per web instructions
(see the prev link), extract the backtrace (using gdb) and share it with me.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
https://www.siphub.com
On 08.12.2023 16:13, Simon Gajski wrote:
Hi Bogdan
you were right
we have the core file now
Please let me know where to send it (gzipped is approx 2MB)
BR
Simon
Bogdan-Andrei Iancu je 08.12.2023 ob 14:17 napisal:
Hi Simon,
Maybe you have some apport service running on the machine, moving the core file.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
https://www.siphub.com
On 08.12.2023 13:35, Neven Grzancic wrote:
Dear Bogdan,
we tried following https://opensips.org/Documentation/TroubleShooting-Crash and
trying different things, and even though the syslog shows that the core file
was dumped:
Dec 8 12:25:13 sbc2 /usr/sbin/opensips[4363]: CRITICAL:core:sig_usr: segfault
in process pid: 4363, id: 7
Dec 8 12:25:13 sbc2 kernel: [ 3394.269733] opensips[4363]: segfault at 0 ip
0000000000000000 sp 00007ffdc458f6a8 error 14 in opensips[55816fbd0000+1f000]
Dec 8 12:25:18 sbc2 mariadbd[922]: 2023-12-08 12:25:18 400 [Warning] Aborted
connection 400 to db: 'opensips' user: 'opensips' host: 'localhost' (Got an
error reading communication packets)
Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4356]: INFO:core:handle_sigs: child
process 4363 exited by a signal 11
Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4356]: INFO:core:handle_sigs: core was
generated
Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4356]: INFO:core:handle_sigs:
terminating due to SIGCHLD
Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4359]: INFO:core:sig_usr: signal 15
received
Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4360]: INFO:core:sig_usr: signal 15
received
the core does not get generated, or at least not in the directory provided via
options at runtime (-w /tmp). The issue is always repeatable (the daemon
crashes after every finished call), if there are more active calls at the
moment of the crash, whose calls remain active and proceed normally until
termination (which again causes a crash). We also tried enabling debug mode,
but in that case the process does not start successfully and runs into a
timeout. In any case, any additional help or suggestion on how to successfully
generate the core dump would be much appreciated, as we seem to have ran into a
wall here...
On 08.12.23 12:26, Simon Gajski wrote:
-------- Posredovano sporočilo --------
Zadeva: Re: [OpenSIPS-Users] opensips is restarting after call is finished
Datum: Thu, 7 Dec 2023 17:52:37 +0200
Od: Bogdan-Andrei Iancu <[email protected]>
Za: Simon Gajski <[email protected]>, OpenSIPS users mailling list
<[email protected]>
Hi Simon,
Well, the log is very self explanatory:
Dec 7 15:59:47 sbc2 /usr/sbin/opensips[28488]: INFO:core:handle_sigs: child
process 28495 exited by a signal 11
Dec 7 15:59:47 sbc2 /usr/sbin/opensips[28488]: INFO:core:handle_sigs: core was
generated
Dec 7 15:59:47 sbc2 /usr/sbin/opensips[28488]: INFO:core:handle_sigs:
terminating due to SIGCHLD
Or shortly, you opensips just crashed :). So see
https://opensips.org/Documentation/TroubleShooting-Crash for how to report
further.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
https://www.siphub.com
On 07.12.2023 17:17, Simon Gajski via Users wrote:
Hi
we are running opensips 3.4.2
on Ubuntu 22.04.3 LTS
and also use RTPengine Version: 10.5.0.0+0~mr10.5.0.0 git-master-74075f63
Opensips acts as SBC with RTP engine enabled.
Calls are working fine, however after each call is finished, bellow action
happens.
We had same problem with opensips 3.2 and Ubuntu 20. So we did upgrade to
latest stable release, and it is the same.
Is this opensips script configuration issue or a bug? And how could I fix it?
[....]
Thanks for all tips.
BR
Simon
_______________________________________________
Users mailing list
[email protected]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
--
Best regards / Lep pozdrav
Neven Grzancic, dipl.inž.el.(UN)
techsupport - systems administration & networking
-----------------------------------------------------------------------
Softnet d.o.o. tel: +386 1 8100 100 | the quieter you
Borovec 2 direct: +386 1 8100 107 | become, the more you
1236 Trzin fax: +386 1 8100 101 | are able to hear
-----------------------------------------------------------------------
_______________________________________________
Users mailing list
[email protected]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
