Package: asterisk Version: 1:13.14.1~dfsg-2+deb9u3 Severity: important Tags: upstream
Dear Maintainer, I'm using Asterisk with its PJSIP backend. Every few hours Asterisk segfaults in PJSIP library code. According to backtraces of coredumps the segfaults seem to be related to SIP registration handling. I cannot say where the root cause is, so I'm reporting this against asterisk and not the PJSIP library. To work around this problem I'm currently using a self-built version of upstream Asterisk (built-in PJSIP). From this experience I can say, that upstream version 13.15.0 does NOT have the described problem (not a single segfault over months). However I would really like to use standard Debian stable packages, without self-built stuff. Details: Over the course of roughly 24h hours I recently got 13 segfaults. 6 of these segfaults occured in a function called tx_data_destroy() in libpjsip: #0 tx_data_destroy (tdata=<optimized out>) at ../src/pjsip/sip_transport.c:485 485 pjsip_endpt_release_pool( tdata->mgr->endpt, tdata->pool ); (gdb) bt #0 tx_data_destroy (tdata=<optimized out>) at ../src/pjsip/sip_transport.c:485 #1 0x00007f686cb59cc8 in pjsip_tx_data_dec_ref (tdata=0x7f6814005748) at ../src/pjsip/sip_transport.c:501 #2 0x00007f67b22b5740 in registration_response_destroy (obj=0x7f685c000dc0) at res_pjsip_outbound_registration.c:741 #3 0x000055ac1cbe7f39 in internal_ao2_ref (user_data=user_data@entry=0x7f685c000dc0, delta=delta@entry=-1, file=file@entry=0x55ac1cd4e066 "astobj2.c", line=line@entry=518, func=func@entry=0x55ac1cd4e158 <__FUNCTION__.9326> "__ao2_ref") at astobj2.c:451 #4 0x000055ac1cbe8528 in __ao2_ref (user_data=user_data@entry=0x7f685c000dc0, delta=delta@entry=-1) at astobj2.c:518 #5 0x00007f67b22b6ffa in handle_registration_response (data=0x7f685c000dc0) at res_pjsip_outbound_registration.c:825 #6 0x000055ac1cd290e8 in ast_taskprocessor_execute (tps=tps@entry=0x55ac1e968ff0) at taskprocessor.c:965 #7 0x000055ac1cd310a0 in execute_tasks (data=0x55ac1e968ff0) at threadpool.c:1322 #8 0x000055ac1cd290e8 in ast_taskprocessor_execute (tps=0x55ac1e39b2c0) at taskprocessor.c:965 #9 0x000055ac1cd30a74 in threadpool_execute (pool=0x55ac1e39ae80) at threadpool.c:351 #10 worker_active (worker=0x7f67e0001a30) at threadpool.c:1105 #11 worker_start (arg=arg@entry=0x7f67e0001a30) at threadpool.c:1024 #12 0x000055ac1cd3908c in dummy_start (data=<optimized out>) at utils.c:1235 #13 0x00007f687358a494 in start_thread (arg=0x7f686e2ae700) at pthread_create.c:333 #14 0x00007f6872194acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 (gdb) list 480 pj_lock_release(tdata->mgr->lock); 481 #endif 482 483 pj_atomic_destroy( tdata->ref_cnt ); 484 pj_lock_destroy( tdata->lock ); 485 pjsip_endpt_release_pool( tdata->mgr->endpt, tdata->pool ); 486 } (gdb) disassemble Dump of assembler code for function tx_data_destroy: 0x00007f686cb59c20 <+0>: push %rbx 0x00007f686cb59c21 <+1>: mov %rdi,%rbx 0x00007f686cb59c24 <+4>: callq 0x7f686cb482d0 <pj_log_get_level@plt> 0x00007f686cb59c29 <+9>: cmp $0x4,%eax 0x00007f686cb59c2c <+12>: jle 0x7f686cb59c4b <tx_data_destroy+43> 0x00007f686cb59c2e <+14>: mov %rbx,%rdi 0x00007f686cb59c31 <+17>: callq 0x7f686cb48c70 <pjsip_tx_data_get_info@plt> 0x00007f686cb59c36 <+22>: lea 0x18(%rbx),%rdi 0x00007f686cb59c3a <+26>: lea 0x16701(%rip),%rsi # 0x7f686cb70342 0x00007f686cb59c41 <+33>: mov %rax,%rdx 0x00007f686cb59c44 <+36>: xor %eax,%eax 0x00007f686cb59c46 <+38>: callq 0x7f686cb48100 <pj_log_5@plt> 0x00007f686cb59c4b <+43>: lea 0x3a8(%rbx),%rdi 0x00007f686cb59c52 <+50>: callq 0x7f686cb48b10 <pjsip_tpselector_dec_ref@plt> 0x00007f686cb59c57 <+55>: mov 0x1b0(%rbx),%rdi 0x00007f686cb59c5e <+62>: callq 0x7f686cb48400 <pj_atomic_destroy@plt> 0x00007f686cb59c63 <+67>: mov 0x180(%rbx),%rdi 0x00007f686cb59c6a <+74>: callq 0x7f686cb48870 <pj_lock_destroy@plt> 0x00007f686cb59c6f <+79>: mov 0x50(%rbx),%rax 0x00007f686cb59c73 <+83>: mov 0x10(%rbx),%rsi 0x00007f686cb59c77 <+87>: pop %rbx => 0x00007f686cb59c78 <+88>: mov 0x10(%rax),%rdi 0x00007f686cb59c7c <+92>: jmpq 0x7f686cb48be0 <pjsip_endpt_release_pool@plt> End of assembler dump. (gdb) up #1 0x00007f686cb59cc8 in pjsip_tx_data_dec_ref (tdata=0x7f6814005748) at ../src/pjsip/sip_transport.c:501 501 tx_data_destroy(tdata); (gdb) print tdata $1 = (pjsip_tx_data *) 0x7f6814005748 (gdb) print tdata->pool $2 = (pj_pool_t *) 0x7f6814005645 (gdb) print tdata->mgr $3 = (pjsip_tpmgr *) 0x554b43415250 (gdb) print tdata->mgr->endpt Cannot access memory at address 0x554b43415260 It seems like the endpoint struct is gone? But why? Broken pointer? Already free'ed? Here are the other types of segfaults, which I haven't had a closer look at yet: 2 segfaults occured in function pj_atomic_inc_and_get() in libpj: (gdb) bt #0 0x00007fce2dcd4999 in pj_atomic_inc_and_get () from /usr/lib/x86_64-linux-gnu/libpj.so.2 #1 0x00007fcdb878e5a3 in sip_outbound_registration_response_cb (param=0x7fce7467c6e0) at res_pjsip_outbound_registration.c:956 #2 0x00007fce2f250358 in ?? () from /usr/lib/x86_64-linux-gnu/libpjsip-ua.so.2 #3 0x00007fce2f251a2f in ?? () from /usr/lib/x86_64-linux-gnu/libpjsip-ua.so.2 #4 0x00007fce2ee0bb11 in tsx_set_state (tsx=tsx@entry=0x7fce34005988, state=state@entry=PJSIP_TSX_STATE_COMPLETED, event_src_type=event_src_type@entry=PJSIP_EVENT_RX_MSG, event_src=0x7fce180098e8, flag=flag@entry=0) at ../src/pjsip/sip_transaction.c:1234 #5 0x00007fce2ee0d550 in tsx_on_state_proceeding_uac (tsx=0x7fce34005988, event=0x7fce7467ca80) at ../src/pjsip/sip_transaction.c:2958 #6 0x00007fce2ee0d76e in tsx_on_state_calling (tsx=0x7fce34005988, event=0x7fce7467ca80) at ../src/pjsip/sip_transaction.c:2541 #7 0x00007fce2ee0eaef in pjsip_tsx_recv_msg (tsx=tsx@entry=0x7fce34005988, rdata=rdata@entry=0x7fce180098e8) at ../src/pjsip/sip_transaction.c:1788 #8 0x00007fce2ee0ebb5 in mod_tsx_layer_on_rx_response (rdata=0x7fce180098e8) at ../src/pjsip/sip_transaction.c:876 #9 0x00007fce2edf93e6 in pjsip_endpt_process_rx_data (endpt=<optimized out>, rdata=rdata@entry=0x7fce180098e8, p=p@entry=0x7fcded3a8be0 <param>, p_handled=p_handled@entry=0x7fce7467cb94) at ../src/pjsip/sip_endpoint.c:895 #10 0x00007fcded1851bc in distribute (data=0x7fce180098e8) at res_pjsip/pjsip_distributor.c:769 #11 0x000055ef7b8b30e8 in ast_taskprocessor_execute (tps=tps@entry=0x55ef7c9b0e40) at taskprocessor.c:965 #12 0x000055ef7b8bb0a0 in execute_tasks (data=0x55ef7c9b0e40) at threadpool.c:1322 #13 0x000055ef7b8b30e8 in ast_taskprocessor_execute (tps=0x55ef7c101420) at taskprocessor.c:965 #14 0x000055ef7b8baa74 in threadpool_execute (pool=0x55ef7c0ffde0) at threadpool.c:351 #15 worker_active (worker=0x7fcddc00b9e0) at threadpool.c:1105 #16 worker_start (arg=arg@entry=0x7fcddc00b9e0) at threadpool.c:1024 #17 0x000055ef7b8c308c in dummy_start (data=<optimized out>) at utils.c:1235 #18 0x00007fce79959494 in start_thread (arg=0x7fce7467d700) at pthread_create.c:333 #19 0x00007fce78563acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 2 segfaults occured in function pj_pool_alloc_from_block() in libpj: (gdb) bt #0 0x00007fdde6efcea4 in pj_pool_alloc_from_block () from /usr/lib/x86_64-linux-gnu/libpj.so.2 #1 0x00007fdde6efd105 in pj_pool_alloc () from /usr/lib/x86_64-linux-gnu/libpj.so.2 #2 0x00007fdde6efd145 in pj_pool_calloc () from /usr/lib/x86_64-linux-gnu/libpj.so.2 #3 0x00007fddfc0550d3 in pj_pool_zalloc (size=288, pool=pool@entry=0x7fdd6400ca18) at ../../pjlib/include/pj/pool.h:476 #4 pjsip_authorization_hdr_create (pool=pool@entry=0x7fdd6400ca18) at ../src/pjsip/sip_auth_msg.c:51 #5 0x00007fddfc0532b0 in auth_respond (req_pool=req_pool@entry=0x7fdd6400ca18, hdr=hdr@entry=0x7fddb8008c00, uri=uri@entry=0x7fdd6400ced0, cred_info=0x7fdd6400c998, method=0x7fdd6400d408, sess_pool=<optimized out>, cached_auth=0x7fdd6400ca48, p_h_auth=0x7fdd3ac205e8) at ../src/pjsip/sip_auth_client.c:694 #6 0x00007fddfc05425c in process_auth (h_auth=0x7fdd3ac205e8, cached_auth=0x7fdd6400ca48, sess=0x7fdd3ac20650, tdata=0x7fdd6400c998, uri=0x7fdd6400ced0, hchal=0x7fddb8008c00, req_pool=0x7fdd6400ca18) at ../src/pjsip/sip_auth_client.c:1115 #7 pjsip_auth_clt_reinit_req (sess=sess@entry=0x7fdd3ac20650, rdata=rdata@entry=0x7fddb8007218, old_request=old_request@entry=0x7fdd6400c998, new_request=new_request@entry=0x7fdd3ac20728) at ../src/pjsip/sip_auth_client.c:1200 #8 0x00007fdd41aa6009 in digest_create_request_with_auth_from_old (auths=<optimized out>, challenge=0x7fddb8007218, old_request=0x7fdd6400c998, new_request=0x7fdd3ac20728) at res_pjsip_outbound_authenticator_digest.c:126 #9 0x00007fdd4189f162 in handle_registration_response (data=0x7fddb8003640) at res_pjsip_outbound_registration.c:811 #10 0x00005651df0160e8 in ast_taskprocessor_execute (tps=tps@entry=0x5651dfdd1690) at taskprocessor.c:965 #11 0x00005651df01e0a0 in execute_tasks (data=0x5651dfdd1690) at threadpool.c:1322 #12 0x00005651df0160e8 in ast_taskprocessor_execute (tps=0x5651e02389b0) at taskprocessor.c:965 #13 0x00005651df01da74 in threadpool_execute (pool=0x5651e0237f30) at threadpool.c:351 #14 worker_active (worker=0x7fdd6c000a80) at threadpool.c:1105 #15 worker_start (arg=arg@entry=0x7fdd6c000a80) at threadpool.c:1024 #16 0x00005651df02608c in dummy_start (data=<optimized out>) at utils.c:1235 #17 0x00007fde02a7a494 in start_thread (arg=0x7fdd3ac21700) at pthread_create.c:333 #18 0x00007fde01684acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 2 segfaults occured in function process_auth() in libpjsip: (gdb) bt #0 0x00007f7858a28fab in process_auth (h_auth=0x7f77d83435e8, cached_auth=0x7f7804004698, sess=0x7f77d8343650, tdata=0x7f78040045e8, uri=0x7f7804004b20, hchal=0x7f78200086e0, req_pool=0x7f7804004668) at ../src/pjsip/sip_auth_client.c:1025 #1 pjsip_auth_clt_reinit_req (sess=sess@entry=0x7f77d8343650, rdata=rdata@entry=0x7f7820006cf8, old_request=old_request@entry=0x7f78040045e8, new_request=new_request@entry=0x7f77d8343728) at ../src/pjsip/sip_auth_client.c:1200 #2 0x00007f779e4be009 in digest_create_request_with_auth_from_old (auths=<optimized out>, challenge=0x7f7820006cf8, old_request=0x7f78040045e8, new_request=0x7f77d8343728) at res_pjsip_outbound_authenticator_digest.c:126 #3 0x00007f779e2b7162 in handle_registration_response (data=0x7f7820001c80) at res_pjsip_outbound_registration.c:811 #4 0x00005604f0d320e8 in ast_taskprocessor_execute (tps=tps@entry=0x5604f28b3390) at taskprocessor.c:965 #5 0x00005604f0d3a0a0 in execute_tasks (data=0x5604f28b3390) at threadpool.c:1322 #6 0x00005604f0d320e8 in ast_taskprocessor_execute (tps=0x5604f29697c0) at taskprocessor.c:965 #7 0x00005604f0d39a74 in threadpool_execute (pool=0x5604f2967e00) at threadpool.c:351 #8 worker_active (worker=0x7f77cc0055a0) at threadpool.c:1105 #9 worker_start (arg=arg@entry=0x7f77cc0055a0) at threadpool.c:1024 #10 0x00005604f0d4208c in dummy_start (data=<optimized out>) at utils.c:1235 #11 0x00007f785f44f494 in start_thread (arg=0x7f77d8344700) at pthread_create.c:333 #12 0x00007f785e059acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 I'm not familier and have never hacked on Asterisk/PJSIP code yet, so I'm happy to be guided by someone who knows the code and where to look. -- System Information: Debian Release: 9.5 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.110 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Init: sysvinit (via /sbin/init) Versions of packages asterisk depends on: ii adduser 3.115 ii asterisk-config 1:13.14.1~dfsg-2+deb9u3 ii asterisk-core-sounds-en 1.4.27-1 ii asterisk-modules 1:13.14.1~dfsg-2+deb9u3 ii init-system-helpers 1.48 ii libbsd0 0.8.3-1 ii libc6 2.24-11+deb9u3 ii libcap2 1:2.25-1 ii libedit2 3.1-20160903-3 ii libgcc1 1:6.3.0-18+deb9u1 ii libjansson4 2.9-1 ii libncurses5 6.0+20161126-1+deb9u2 ii libpopt0 1.16-10+b2 ii libsqlite3-0 3.16.2-5+deb9u1 ii libssl1.1 1.1.0f-3+deb9u2 ii libstdc++6 6.3.0-18+deb9u1 ii libsystemd0 232-23 ii libtinfo5 6.0+20161126-1+deb9u2 ii liburiparser1 0.8.4-1 ii libuuid1 2.29.2-1+deb9u1 ii libxml2 2.9.4+dfsg1-2.2+deb9u2 ii libxslt1.1 1.1.29-2.1 ii lsb-base 9.20161125 Versions of packages asterisk recommends: ii asterisk-moh-opsound-gsm 2.03-1 ii asterisk-voicemail-imapstorage [asterisk-voicemail 1:13.14.1~dfsg-2+deb9u3 ii sox 14.4.1-5+b2 Versions of packages asterisk suggests: pn asterisk-dahdi <none> pn asterisk-dev <none> pn asterisk-doc <none> pn asterisk-ooh323 <none> pn asterisk-opus <none> pn asterisk-vpb <none> -- no debconf information