[jira] [Commented] (TS-4195) out of traffic_manager causes a double free in traffic_server
[
https://issues.apache.org/jira/browse/TS-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15464450#comment-15464450
]
Dimitry Andric commented on TS-4195:
Possibly related to the fix for TS-3863.
> out of traffic_manager causes a double free in traffic_server
> -
>
> Key: TS-4195
> URL: https://issues.apache.org/jira/browse/TS-4195
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
>Reporter: Leif Hedstrom
>Assignee: Bryan Call
>Priority: Blocker
> Fix For: 7.0.0
>
>
> While testing stuff, I was running traffic_manager from command line, and
> then I get a crash from traffic_server:
> {code}
> root@loki 407/0 # ./bin/traffic_manager
> [E. Mgmt] log ==> [TrafficManager] using root directory '/opt/ats'
> traffic_server: using root directory '/opt/ats'
> ^C[TrafficManager] ==> Cleaning up and reissuing signal #2
> traffic_server: Interrupt (Signal sent by the kernel 0 0)
> 9083 sent by kill()*** Error in `/opt/ats/bin/traffic_server': corrupted
> double-linked list: 0x028f8940 ***
> === Backtrace: =
> /lib64/libc.so.6(+0x77da5)[0x2ad58f3fcda5]
> /lib64/libc.so.6(+0x80c06)[0x2ad58f405c06]
> /lib64/libc.so.6(cfree+0x4c)[0x2ad58f408cac]
> /lib64/libc.so.6(+0x39685)[0x2ad58f3be685]
> /lib64/libc.so.6(+0x396a5)[0x2ad58f3be6a5]
> /opt/ats/bin/traffic_server[0x4e300atraffic_server: Segmentation fault
> (Address not mapped to object [0x55b02140])
> traffic_server - STACK TRACE:
> /lib64/libc.so.6(nanosleep+0x2d)[0x2ad58f44d7ad]
> /opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4abece]
> /lib64/libpthread.so.0(+0x109f0)[0x2ad58e3709f0]
> /lib64/libc.so.6(sleep+0xd4)[0x2ad58f44d644]
> /opt/ats/bin/traffic_server(_Z19startProcessManagerPv+0xb1)[0x69b8a1]
> /lib64/libpthread.so.0(+0x760a)[0x2ad58e36760a]
> /lib64/libc.so.6(clone+0x6d)[0x2ad58f487a4d]
> === Memory map:
> /lib64/libc.so.6(+0x395ad)[0x2ad58f3be5ad]
> 0040-008a6000 r-xp 00:24 1775473
> /opt/ats/bin/traffic_server
> 00aa6000-00ab3000 r--p 004a6000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab3000-00ab9000 rw-p 004b3000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab9000-01097000 rw-p 00:00 0
> 028dd000-02cb9000 rw-p 00:00 0
> [heap]
> 2ad58c52c000-2ad58c54d000 r-xp 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c54d000-2ad58c55 rw-p 00:00 0
> 2ad58c55-2ad58c56 rwxp 00:00 0
> 2ad58c56b000-2ad58c6ed000 rw-p 00:00 0
> 2ad58c6ed000-2ad58c6fd000 rwxp 00:00 0
> 2ad58c6fd000-2ad58c748000 rw-p 00:00 0
> 2ad58c74c000-2ad58c74d000 r--p 0002 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c74d000-2ad58c74e000 rw-p 00021000 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c74e000-2ad58c74f000 rw-p 00:00 0
> 2ad58c74f000-2ad58c79 r-xp 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c79-2ad58c99 ---p 00041000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c99-2ad58c991000 r--p 00041000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c991000-2ad58c993000 rw-p 00042000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c993000-2ad58c994000 rw-p 00:00 0
> 2ad58c994000-2ad58c9cb000 r-xp 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58c9cb000-2ad58cbcb000 ---p 00037000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcb000-2ad58cbcc000 r--p 00037000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcc000-2ad58cbcd000 rw-p 00038000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcd000-2ad58cd77000 r-xp 00:24 1441754
> /usr/lib64/libtcl8.6.so
> 2ad58cd77000-2ad58cf77000 ---p 001aa000 00:24 1441754
> /lib64/libc.so.6( /usr/lib64/libtcl8.6.so
> 2ad58cf77000-2ad58cf86000 r--p 001aa000 00:24 1441754
> /usr/lib64/libtcl8.6.so
> 2ad58cf86000-2ad58cf87000 rw-p 001b9000 00:24 1441754
> /usr/lib64/libtcl8.6.so
> 2ad58cf87000-2ad58cf88000 rw-p 00:00 0
> 2ad58cf88000-2ad58cf9f000 r-xp 00:24 1389936
> /usr/lib64/libresolv-2.22.so
> 2ad58cf9f000-2ad58d19f000 ---p 00017000 00:24 1389936
> /usr/lib64/libresolv-2.22.so
> 2ad58d19f000-2ad58d1a r--p 00017000 00:24 1389936
> /usr/lib64/libresolv-2.22.so
> 2ad58d1a-2ad58d1a1000 rw-p 00018000 00:24 1389936
[jira] [Commented] (TS-4195) out of traffic_manager causes a double free in traffic_server
[
https://issues.apache.org/jira/browse/TS-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455447#comment-15455447
]
Dimitry Andric commented on TS-4195:
For me, this starts occurring after [commit
d8bc508|https://git-dual.apache.org/repos/asf?p=trafficserver.git;a=commitdiff;h=d8bc5089b265574adf2d1ea3d904823eb9c4c30a]
("TS-3863: Add support for ASAN leak detection"). This introduces a
{{sleep(1)}} in {{proxy_signal_handler()}}, and also changes the {{_exit()}}
call to {{exit()}}. Sleeping in a signal handler is never OK, and calling
{{exit()}} is also dangerous. There is a risk that you get a signal while you
sleeping, or while {{exit()}} itself is running, and then all bets are off.
In any case, if I locally revert [commit
d8bc508|https://git-dual.apache.org/repos/asf?p=trafficserver.git;a=commitdiff;h=d8bc5089b265574adf2d1ea3d904823eb9c4c30a],
for me the crashes seem to disappear completely.
> out of traffic_manager causes a double free in traffic_server
> -
>
> Key: TS-4195
> URL: https://issues.apache.org/jira/browse/TS-4195
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
>Reporter: Leif Hedstrom
>Assignee: Bryan Call
>Priority: Blocker
> Fix For: 7.0.0
>
>
> While testing stuff, I was running traffic_manager from command line, and
> then I get a crash from traffic_server:
> {code}
> root@loki 407/0 # ./bin/traffic_manager
> [E. Mgmt] log ==> [TrafficManager] using root directory '/opt/ats'
> traffic_server: using root directory '/opt/ats'
> ^C[TrafficManager] ==> Cleaning up and reissuing signal #2
> traffic_server: Interrupt (Signal sent by the kernel 0 0)
> 9083 sent by kill()*** Error in `/opt/ats/bin/traffic_server': corrupted
> double-linked list: 0x028f8940 ***
> === Backtrace: =
> /lib64/libc.so.6(+0x77da5)[0x2ad58f3fcda5]
> /lib64/libc.so.6(+0x80c06)[0x2ad58f405c06]
> /lib64/libc.so.6(cfree+0x4c)[0x2ad58f408cac]
> /lib64/libc.so.6(+0x39685)[0x2ad58f3be685]
> /lib64/libc.so.6(+0x396a5)[0x2ad58f3be6a5]
> /opt/ats/bin/traffic_server[0x4e300atraffic_server: Segmentation fault
> (Address not mapped to object [0x55b02140])
> traffic_server - STACK TRACE:
> /lib64/libc.so.6(nanosleep+0x2d)[0x2ad58f44d7ad]
> /opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4abece]
> /lib64/libpthread.so.0(+0x109f0)[0x2ad58e3709f0]
> /lib64/libc.so.6(sleep+0xd4)[0x2ad58f44d644]
> /opt/ats/bin/traffic_server(_Z19startProcessManagerPv+0xb1)[0x69b8a1]
> /lib64/libpthread.so.0(+0x760a)[0x2ad58e36760a]
> /lib64/libc.so.6(clone+0x6d)[0x2ad58f487a4d]
> === Memory map:
> /lib64/libc.so.6(+0x395ad)[0x2ad58f3be5ad]
> 0040-008a6000 r-xp 00:24 1775473
> /opt/ats/bin/traffic_server
> 00aa6000-00ab3000 r--p 004a6000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab3000-00ab9000 rw-p 004b3000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab9000-01097000 rw-p 00:00 0
> 028dd000-02cb9000 rw-p 00:00 0
> [heap]
> 2ad58c52c000-2ad58c54d000 r-xp 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c54d000-2ad58c55 rw-p 00:00 0
> 2ad58c55-2ad58c56 rwxp 00:00 0
> 2ad58c56b000-2ad58c6ed000 rw-p 00:00 0
> 2ad58c6ed000-2ad58c6fd000 rwxp 00:00 0
> 2ad58c6fd000-2ad58c748000 rw-p 00:00 0
> 2ad58c74c000-2ad58c74d000 r--p 0002 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c74d000-2ad58c74e000 rw-p 00021000 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c74e000-2ad58c74f000 rw-p 00:00 0
> 2ad58c74f000-2ad58c79 r-xp 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c79-2ad58c99 ---p 00041000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c99-2ad58c991000 r--p 00041000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c991000-2ad58c993000 rw-p 00042000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c993000-2ad58c994000 rw-p 00:00 0
> 2ad58c994000-2ad58c9cb000 r-xp 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58c9cb000-2ad58cbcb000 ---p 00037000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcb000-2ad58cbcc000 r--p 00037000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcc000-2ad58cbcd000 rw-p 00038000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcd000-2ad58cd77000 r-xp 00:24 1441754
> /usr/lib64/libtcl8.6.so
> 2ad58cd77000-2ad58cf77000 ---p 00
[jira] [Commented] (TS-4195) out of traffic_manager causes a double free in traffic_server
[
https://issues.apache.org/jira/browse/TS-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431872#comment-15431872
]
Bryan Call commented on TS-4195:
I also see this one:
{noformat}
==23255==ERROR: AddressSanitizer: attempting double-free on 0x61900f80 in
thread T0 ([ET_NET 0]):
#0 0x2b24e55bbac0 in free (/lib64/libasan.so.3+0xc6ac0)
#1 0x2b24e9155214 in __run_exit_handlers (/lib64/libc.so.6+0x39214)
#2 0x2b24e9155234 in __GI_exit (/lib64/libc.so.6+0x39234)
#3 0x587239 in proxy_signal_handler
/home/bcall/dev/apache/trafficserver/proxy/Main.cc:409
#4 0x2b24e80fac2f (/lib64/libpthread.so.0+0x10c2f)
#5 0x2b24e921f4b2 in __GI_epoll_wait (/lib64/libc.so.6+0x1034b2)
#6 0xc31421 in NetHandler::mainNetEvent(int, Event*)
/home/bcall/dev/apache/trafficserver/iocore/net/UnixNet.cc:423
#7 0xd13ce0 in Continuation::handleEvent(int, void*)
/home/bcall/dev/apache/trafficserver/iocore/eventsystem/I_Continuation.h:153
#8 0xd13ce0 in EThread::process_event(Event*, int)
/home/bcall/dev/apache/trafficserver/iocore/eventsystem/UnixEThread.cc:148
#9 0xd16bc6 in EThread::execute()
/home/bcall/dev/apache/trafficserver/iocore/eventsystem/UnixEThread.cc:275
#10 0x49ac50 in main /home/bcall/dev/apache/trafficserver/proxy/Main.cc:1956
#11 0x2b24e913c730 in __libc_start_main (/lib64/libc.so.6+0x20730)
#12 0x4aa598 in _start (/usr/local/bin/traffic_server+0x4aa598)
0x61900f80 is located 0 bytes inside of 1040-byte region
[0x61900f80,0x61901390)
freed by thread T1 here:
=
#0 0x2b24e55bbac0 in free (/lib64/libasan.so.3+0xc6ac0)
#1 0x2b24e9155214 in __run_exit_handlers (/lib64/libc.so.6+0x39214)
==23255==ERROR: LeakSanitizer: detected memory leaks
previously allocated by thread T0 ([ET_NET 0]) here:
Direct leak of 257 byte(s) in 1 object(s) allocated from:
#0 0x2b24e55bbfe0 in calloc (/lib64/libasan.so.3+0xc6fe0)
#1 0x2b24e91553e8 in __new_exitfn (/lib64/libc.so.6+0x393e8)
#0 0x2b24e55bbe20 in malloc (/lib64/libasan.so.3+0xc6e20)
Thread T1 created by T0 ([ET_NET 0]) here:
#1 0x2b24e64f30d5 in ats_malloc
/home/bcall/dev/apache/trafficserver/lib/ts/ink_memory.cc:59
#2 0x4988d5 in ats_scoped_str::ats_scoped_str(unsigned long)
../lib/ts/ink_memory.h:442
#3 0x4988d5 in main /home/bcall/dev/apache/trafficserver/proxy/Main.cc:1608
#0 0x2b24e5526458 in pthread_create (/lib64/libasan.so.3+0x31458)
#4 0x2b24e913c730 in __libc_start_main (/lib64/libc.so.6+0x20730)
#1 0x49833c in ink_thread_create ../lib/ts/ink_thread.h:147
#2 0x49833c in ProcessManager::start() ../mgmt/ProcessManager.h:65
#3 0x49833c in initialize_process_manager
/home/bcall/dev/apache/trafficserver/proxy/Main.cc:503
#4 0x49833c in main /home/bcall/dev/apache/trafficserver/proxy/Main.cc:1567
Direct leak of 40 byte(s) in 1 object(s) allocated from:
#5 0x2b24e913c730 in __libc_start_main (/lib64/libc.so.6+0x20730)
#0 0x2b24e55bbe20 in malloc (/lib64/libasan.so.3+0xc6e20)
SUMMARY: AddressSanitizer: double-free (/lib64/libasan.so.3+0xc6ac0) in free
#1 0x2b24e64f30d5 in ats_malloc
/home/bcall/dev/apache/trafficserver/lib/ts/ink_memory.cc:59
==23255==ABORTING
#2 0xd13640 in Thread::start(char const*, unsigned long, void* (*)(void*),
void*) /home/bcall/dev/apache/trafficserver/iocore/eventsystem/Thread.cc:92
[TrafficManager] ==> signal #2
{noformat}
> out of traffic_manager causes a double free in traffic_server
> -
>
> Key: TS-4195
> URL: https://issues.apache.org/jira/browse/TS-4195
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
>Reporter: Leif Hedstrom
>Assignee: Bryan Call
>Priority: Blocker
> Fix For: 7.0.0
>
>
> While testing stuff, I was running traffic_manager from command line, and
> then I get a crash from traffic_server:
> {code}
> root@loki 407/0 # ./bin/traffic_manager
> [E. Mgmt] log ==> [TrafficManager] using root directory '/opt/ats'
> traffic_server: using root directory '/opt/ats'
> ^C[TrafficManager] ==> Cleaning up and reissuing signal #2
> traffic_server: Interrupt (Signal sent by the kernel 0 0)
> 9083 sent by kill()*** Error in `/opt/ats/bin/traffic_server': corrupted
> double-linked list: 0x028f8940 ***
> === Backtrace: =
> /lib64/libc.so.6(+0x77da5)[0x2ad58f3fcda5]
> /lib64/libc.so.6(+0x80c06)[0x2ad58f405c06]
> /lib64/libc.so.6(cfree+0x4c)[0x2ad58f408cac]
> /lib64/libc.so.6(+0x39685)[0x2ad58f3be685]
> /lib64/libc.so.6(+0x396a5)[0x2ad58f3be6a5]
> /opt/ats/bin/traffic_server[0x4e300atraffic_server: Segmentation fault
> (Address not mapped to object [0x55b02140])
> traffic_server - STACK TRACE:
> /lib64/libc.so.6(nanosleep+0x2d)[0x2ad58f44d7ad]
>
[jira] [Commented] (TS-4195) out of traffic_manager causes a double free in traffic_server
[
https://issues.apache.org/jira/browse/TS-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330112#comment-15330112
]
Steven Feltner commented on TS-4195:
CentOS6 compiled with devtoolset-3 (gcc-4.9.2)
I see a similar situation on every exit when I shutdown traffic server via
/etc/init.d/trafficserver stop:
{code}
[Jun 14 11:04:46.886] Manager {0x7f88858bb800} DEBUG: (lm) [TrafficManager] ==>
Sending signal event '10009' payload=2712
[Jun 14 11:04:46.886] Manager {0x7f88858bb800} DEBUG: (lm) [TrafficManager] ==>
Sending signal event '10009' payload=264
traffic_server: Terminated (Signal sent by kill() 11730 0)traffic_server:
Terminated (Signal sent by kill() 11730 0)traffic_server: Terminated (Signal
sent by kill() 11732 0)traffic_server: Terminated (Signal sent by kill() 11732
0)traffic_server: Terminated (Signal sent by kill() 13291 0)*** glibc detected
*** /usr/bin/traffic_server: double free or corruption (!prev):
0x01bcaf60 ***
=== Backtrace: =
/lib64/libc.so.6[0x3f8fa75f4e]
/lib64/libc.so.6[0x3f8fa78cf0]
/lib64/libc.so.6(exit+0x115)[0x3f8fa35b55]
traffic_server: Segmentation fault (Address not mapped to object
[0x39e21e90])traffic_server - STACK TRACE:
/usr/bin/traffic_server[0x537b20]
/usr/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0xc3)[0x5054ed]
/lib64/libc.so.6[0x3f8fa326a0]
/lib64/libc.so.6[0x3f8fa326a0]
/lib64/libc.so.6(epoll_wait+0x33)[0x3f8fae8f33]
/lib64/libc.so.6(exit+0x35)[0x3f8fa35a75]
/usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x173)[0x72e1af]
/usr/bin/traffic_server[0x537b20]
/usr/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50852e]
/lib64/libc.so.6[0x3f8fa326a0]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x136)[0x759090]
/lib64/libc.so.6(epoll_wait+0x33)[0x3f8fae8f33]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x4ae)[0x7596bc]
/usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x173)[0x72e1af]
/usr/bin/traffic_server(main+0x149c)[0x53b376]
/usr/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50852e]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3f8fa1ed5d]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x136)[0x759090]
/usr/bin/traffic_server[0x4ed919]
=== Memory map:
{code}
> out of traffic_manager causes a double free in traffic_server
> -
>
> Key: TS-4195
> URL: https://issues.apache.org/jira/browse/TS-4195
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
>Reporter: Leif Hedstrom
>Assignee: Bryan Call
>Priority: Blocker
> Fix For: 7.0.0
>
>
> While testing stuff, I was running traffic_manager from command line, and
> then I get a crash from traffic_server:
> {code}
> root@loki 407/0 # ./bin/traffic_manager
> [E. Mgmt] log ==> [TrafficManager] using root directory '/opt/ats'
> traffic_server: using root directory '/opt/ats'
> ^C[TrafficManager] ==> Cleaning up and reissuing signal #2
> traffic_server: Interrupt (Signal sent by the kernel 0 0)
> 9083 sent by kill()*** Error in `/opt/ats/bin/traffic_server': corrupted
> double-linked list: 0x028f8940 ***
> === Backtrace: =
> /lib64/libc.so.6(+0x77da5)[0x2ad58f3fcda5]
> /lib64/libc.so.6(+0x80c06)[0x2ad58f405c06]
> /lib64/libc.so.6(cfree+0x4c)[0x2ad58f408cac]
> /lib64/libc.so.6(+0x39685)[0x2ad58f3be685]
> /lib64/libc.so.6(+0x396a5)[0x2ad58f3be6a5]
> /opt/ats/bin/traffic_server[0x4e300atraffic_server: Segmentation fault
> (Address not mapped to object [0x55b02140])
> traffic_server - STACK TRACE:
> /lib64/libc.so.6(nanosleep+0x2d)[0x2ad58f44d7ad]
> /opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4abece]
> /lib64/libpthread.so.0(+0x109f0)[0x2ad58e3709f0]
> /lib64/libc.so.6(sleep+0xd4)[0x2ad58f44d644]
> /opt/ats/bin/traffic_server(_Z19startProcessManagerPv+0xb1)[0x69b8a1]
> /lib64/libpthread.so.0(+0x760a)[0x2ad58e36760a]
> /lib64/libc.so.6(clone+0x6d)[0x2ad58f487a4d]
> === Memory map:
> /lib64/libc.so.6(+0x395ad)[0x2ad58f3be5ad]
> 0040-008a6000 r-xp 00:24 1775473
> /opt/ats/bin/traffic_server
> 00aa6000-00ab3000 r--p 004a6000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab3000-00ab9000 rw-p 004b3000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab9000-01097000 rw-p 00:00 0
> 028dd000-02cb9000 rw-p 00:00 0
> [heap]
> 2ad58c52c000-2ad58c54d000 r-xp 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c54d000-2ad58c55 rw-p 00:00 0
> 2ad58c55-2ad58c56 rwxp 00:00 0
> 2ad58c56b000-2ad58c6ed000 rw-p 00:00 0
> 2ad58c6ed000-2ad58c6fd000 rwxp 00
[jira] [Commented] (TS-4195) out of traffic_manager causes a double free in traffic_server
[
https://issues.apache.org/jira/browse/TS-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328722#comment-15328722
]
James Peach commented on TS-4195:
-
I see this maybe 25% of the time. Here's a suspicious backtrace in SSL code:
{code}
^C[TrafficManager] ==> Cleaning up and reissuing signal #2
traffic_server: Interrupt (Signal sent by the kernel 0 0)
traffic_server: Interrupt (Signal sent by kill() 12962 0)
*** Error in `traffic_server: Segmentation fault (Signal sent by the kernel
[(nil)])
traffic_server - STACK TRACE:
/opt/ats/bin/traffic_server':
/opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xc3)[0x50c999]
/lib64/libpthread.so.0(+0x10a00)[0x2ac70e453a00]
/lib64/libc.so.6(+0x3953d)[0x2ac70f13953d]
/lib64/libc.so.6(+0x39635)[0x2ac70f139635]
/opt/ats/bin/traffic_servertraffic_server: Aborted (Signal sent by tkill()
12968 99)
traffic_server - STACK TRACE:
/opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xc3)[0x50c999]
/lib64/libpthread.so.0(+0x10a00)[0x2ac70e453a00]
/lib64/libc.so.6(gsignal+0x38)[0x2ac70f134a28]
/lib64/libc.so.6(abort+0x16a)[0x2ac70f13662a]
/lib64/libc.so.6(+0x77d7a)[0x2ac70f177d7a]
/lib64/libc.so.6(+0x801ca)[0x2ac70f1801ca]
/lib64/libc.so.6(cfree+0x4c)[0x2ac70f18372c]
/opt/ats/lib/libtsutil.so.7(ats_free+0x28)[0x2ac70c856cea]
[0x53eaa8]
/opt/ats/bin/traffic_server(_Z8ssl_freePv/lib64/libpthread.so.0(+0x10a00)[0x2ac70e453a00]
+0x/lib64/libc.so.618()nanosleep[0x+0x7b92802d]
)[0x2ac70f1c84cd]
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x2ac70d5662ed]
/lib64/libcrypto.so.10(lh_free+0x57)[0x2ac70d61be97]
/lib64/libssl.so.10(SSL_CTX_free+0x7c)[0x2ac70d2c93ec]
/lib64/libc.so.6(sleep+0xd4)[0x2ac70f1c8364]
/opt/ats/bin/traffic_server(_Z14mgmt_sleep_seci+0x15)[0x710e29]
/opt/ats/bin/traffic_server(_Z17SSLReleaseContextP10ssl_ctx_st+0x18)[0x7be54b]
/opt/ats/bin/traffic_server(_ZN15SSLNetProcessor7cleanupEv+0x1c)[0x7af3b6]
/opt/ats/bin/traffic_server(_ZN15SSLNetProcessorD2Ev+0x24)[0x7af6f2]
/opt/ats/bin/traffic_server(_Z19startProcessManagerPv+0x137)[0x70b832]
/lib64/libpthread.so.0(+0x761a)[0x2ac70e44a61a]
/lib64/libc.so.6(clone+0x6d)[0x2ac70f20259d]
/lib64/libc.so.6(+0x395e8)[0x2ac70f1395e8]
/lib64/libc.so.6(+0x39635)[0x2ac70f139635]
[TrafficManager] ==> signal #2
{code}
> out of traffic_manager causes a double free in traffic_server
> -
>
> Key: TS-4195
> URL: https://issues.apache.org/jira/browse/TS-4195
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
>Reporter: Leif Hedstrom
>Assignee: Bryan Call
>Priority: Blocker
> Fix For: 7.0.0
>
>
> While testing stuff, I was running traffic_manager from command line, and
> then I get a crash from traffic_server:
> {code}
> root@loki 407/0 # ./bin/traffic_manager
> [E. Mgmt] log ==> [TrafficManager] using root directory '/opt/ats'
> traffic_server: using root directory '/opt/ats'
> ^C[TrafficManager] ==> Cleaning up and reissuing signal #2
> traffic_server: Interrupt (Signal sent by the kernel 0 0)
> 9083 sent by kill()*** Error in `/opt/ats/bin/traffic_server': corrupted
> double-linked list: 0x028f8940 ***
> === Backtrace: =
> /lib64/libc.so.6(+0x77da5)[0x2ad58f3fcda5]
> /lib64/libc.so.6(+0x80c06)[0x2ad58f405c06]
> /lib64/libc.so.6(cfree+0x4c)[0x2ad58f408cac]
> /lib64/libc.so.6(+0x39685)[0x2ad58f3be685]
> /lib64/libc.so.6(+0x396a5)[0x2ad58f3be6a5]
> /opt/ats/bin/traffic_server[0x4e300atraffic_server: Segmentation fault
> (Address not mapped to object [0x55b02140])
> traffic_server - STACK TRACE:
> /lib64/libc.so.6(nanosleep+0x2d)[0x2ad58f44d7ad]
> /opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4abece]
> /lib64/libpthread.so.0(+0x109f0)[0x2ad58e3709f0]
> /lib64/libc.so.6(sleep+0xd4)[0x2ad58f44d644]
> /opt/ats/bin/traffic_server(_Z19startProcessManagerPv+0xb1)[0x69b8a1]
> /lib64/libpthread.so.0(+0x760a)[0x2ad58e36760a]
> /lib64/libc.so.6(clone+0x6d)[0x2ad58f487a4d]
> === Memory map:
> /lib64/libc.so.6(+0x395ad)[0x2ad58f3be5ad]
> 0040-008a6000 r-xp 00:24 1775473
> /opt/ats/bin/traffic_server
> 00aa6000-00ab3000 r--p 004a6000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab3000-00ab9000 rw-p 004b3000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab9000-01097000 rw-p 00:00 0
> 028dd000-02cb9000 rw-p 00:00 0
> [heap]
> 2ad58c52c000-2ad58c54d000 r-xp 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c54d000-2ad58c55 rw-p 00:00 0
> 2ad58c55-2ad58c56 rwxp 00:00 0
> 2ad58c56b000-2ad58c6ed000 rw-p 00:00 0
> 2ad58c6ed000-2ad58c6fd000 rwxp 00:00 0
> 2ad58c6fd000-2ad58c
