[
https://issues.apache.org/jira/browse/TS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336637#comment-14336637
]
Susan Hinrichs commented on TS-3405:
------------------------------------
Sadly, spoke too soon on the crashing front. I left it running for a while and
then came back to refresh a page and ATS seg faulted right away. This is with
my fetchsm fix to avoid the CPU spin problem, so it is possible that that fix
introduced the crash. In which case, we need to reconsider how FetchSM/HTTP2
should deal with EOS.
Here is the stack track.
{code}
(gdb) bt
#0 0x0000000000649696 in Http2ConnectionState::send_data_frame (
this=0x7fffd8035198, fetch_sm=0x7fffdc7099e0)
at Http2ConnectionState.cc:823
#1 0x0000000000649290 in Http2ConnectionState::restart_streams (
this=0x7fffd8035198) at Http2ConnectionState.cc:749
#2 0x0000000000648609 in rcv_window_update_frame (cs=..., cstate=...,
frame=...) at Http2ConnectionState.cc:473
#3 0x0000000000648ed1 in Http2ConnectionState::main_event_handler (
this=0x7fffd8035198, event=2253, edata=0x7ffff43f47d0)
at Http2ConnectionState.cc:650
#4 0x000000000050cba4 in Continuation::handleEvent (this=0x7fffd8035198,
event=2253, data=0x7ffff43f47d0)
at ../iocore/eventsystem/I_Continuation.h:146
#5 0x000000000064409e in send_connection_event (cont=0x7fffd8035198,
event=2253, edata=0x7ffff43f47d0) at Http2ClientSession.cc:58
#6 0x0000000000645fd7 in Http2ClientSession::state_complete_frame_read (
this=0x7fffd8034f40, event=100, edata=0x7fffe4017fa0)
at Http2ClientSession.cc:383
#7 0x0000000000644d8b in Http2ClientSession::main_event_handler (
this=0x7fffd8034f40, event=100, edata=0x7fffe4017fa0)
at Http2ClientSession.cc:213
#8 0x000000000050cba4 in Continuation::handleEvent (this=0x7fffd8034f40,
event=100, data=0x7fffe4017fa0)
at ../iocore/eventsystem/I_Continuation.h:146
#9 0x0000000000645ceb in Http2ClientSession::state_start_frame_read (
this=0x7fffd8034f40, event=100, edata=0x7fffe4017fa0)
at Http2ClientSession.cc:356
#10 0x0000000000644d8b in Http2ClientSession::main_event_handler (
this=0x7fffd8034f40, event=100, edata=0x7fffe4017fa0)
at Http2ClientSession.cc:213
#11 0x000000000050cba4 in Continuation::handleEvent (this=0x7fffd8034f40,
event=100, data=0x7fffe4017fa0)
at ../iocore/eventsystem/I_Continuation.h:146
#12 0x0000000000786aeb in read_signal_and_update (event=100, vc=0x7fffe4017e80)
at UnixNetVConnection.cc:140
#13 0x0000000000789914 in UnixNetVConnection::readSignalAndUpdate (
this=0x7fffe4017e80, event=100) at UnixNetVConnection.cc:946
#14 0x00000000007701cb in SSLNetVConnection::net_read_io (this=0x7fffe4017e80,
nh=0x7ffff45fb730, lthread=0x7ffff45f8010) at SSLNetVConnection.cc:556
#15 0x000000000078042e in NetHandler::mainNetEvent (this=0x7ffff45fb730,
event=5, e=0x11538a0) at UnixNet.cc:532
#16 0x000000000050cba4 in Continuation::handleEvent (this=0x7ffff45fb730,
event=5, data=0x11538a0) at ../iocore/eventsystem/I_Continuation.h:146
#17 0x00000000007a8b66 in EThread::process_event (this=0x7ffff45f8010,
e=0x11538a0, calling_code=5) at UnixEThread.cc:144
#18 0x00000000007a9187 in EThread::execute (this=0x7ffff45f8010)
at UnixEThread.cc:268
#19 0x00000000007a806c in spawn_thread_internal (a=0x124cb70) at Thread.cc:88
#20 0x000000351e807851 in start_thread () from /lib64/libpthread.so.0
#21 0x000000351e4e890d in clone () from /lib64/libc.so.6
{code}
The variable stream is bogus because fetch_sm is bogus (all set to the
unallocated values).
> Memory use after free in HTTP/2
> -------------------------------
>
> Key: TS-3405
> URL: https://issues.apache.org/jira/browse/TS-3405
> Project: Traffic Server
> Issue Type: Bug
> Components: HTTP/2
> Reporter: Bryan Call
> Fix For: 5.3.0
>
> Attachments: 0002-fix-h2.patch, fetchsm-change.diff, fix-h2.patch
>
>
> From Leif running on docs.trafficserver.apache.org:
>
> {code}
> traffic_server: using root directory '/opt/ats'
> =================================================================
> ==31101==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61800000c888 at pc 0x4f3558 bp 0x2aaf10c88930 sp 0x2aaf10c88928
> READ of size 8 at 0x61800000c888 thread T2 ([ET_NET 1])
> #0 0x4f3557 in Continuation::handleEvent(int, void*)
> ../iocore/eventsystem/I_Continuation.h:146
> #1 0x4f3557 in FetchSM::InvokePluginExt(int)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:301
> #2 0x4f3a7a in FetchSM::process_fetch_read(int)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:465
> #3 0x4f5112 in FetchSM::fetch_handler(int, void*)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:514
> #4 0x59f1b7 in Continuation::handleEvent(int, void*)
> ../iocore/eventsystem/I_Continuation.h:146
> #5 0x59f1b7 in PluginVC::process_read_side(bool)
> /usr/local/src/trafficserver/proxy/PluginVC.cc:640
> #6 0x5abcb9 in PluginVC::main_handler(int, void*)
> /usr/local/src/trafficserver/proxy/PluginVC.cc:206
> #7 0xc821fe in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #8 0xc821fe in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #9 0xc84819 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:238
> #10 0xc80e18 in spawn_thread_internal
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:88
> #11 0x2aaf0b083df2 in start_thread (/lib64/libpthread.so.0+0x7df2)
> #12 0x2aaf0c8ec1ac in clone (/lib64/libc.so.6+0xf61ac)
> 0x61800000c888 is located 8 bytes inside of 816-byte region
> [0x61800000c880,0x61800000cbb0)
> freed by thread T0 ([ET_NET 0]) here:
> #0 0x2aaf08c131c7 in __interceptor_free
> ../../.././libsanitizer/asan/asan_malloc_linux.cc:62
> #1 0x7b7d42 in Http2ClientSession::do_io_close(int)
> /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:194
> #2 0x7b7d42 in Http2ClientSession::main_event_handler(int, void*)
> /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:237
> #3 0xc1351f in Continuation::handleEvent(int, void*)
> ../../iocore/eventsystem/I_Continuation.h:146
> #4 0xc1351f in read_signal_and_update
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140
> #5 0xc1351f in read_signal_done
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185
> #6 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*)
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939
> #7 0xbbabf8 in SSLNetVConnection::net_read_io(NetHandler*, EThread*)
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:596
> #8 0xbda09c in NetHandler::mainNetEvent(int, Event*)
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513
> #9 0xc85089 in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #10 0xc85089 in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #11 0xc85089 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268
> #12 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826
> #13 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> previously allocated by thread T0 ([ET_NET 0]) here:
> #0 0x2aaf08c1393b in __interceptor_posix_memalign
> ../../.././libsanitizer/asan/asan_malloc_linux.cc:130
> #1 0x2aaf09afd2f9 in ats_memalign
> /usr/local/src/trafficserver/lib/ts/ink_memory.cc:96
> #2 0x7cd804 in ClassAllocator<Http2ClientSession>::alloc()
> ../../lib/ts/Allocator.h:124
> #3 0x7cd804 in Http2SessionAccept::accept(NetVConnection*, MIOBuffer*,
> IOBufferReader*)
> /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:57
> #4 0x7cd3c4 in Http2SessionAccept::mainEvent(int, void*)
> /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:69
> #5 0xbc2fae in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*)
> /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:101
> #6 0xc1351f in Continuation::handleEvent(int, void*)
> ../../iocore/eventsystem/I_Continuation.h:146
> #7 0xc1351f in read_signal_and_update
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140
> #8 0xc1351f in read_signal_done
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185
> #9 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*)
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939
> #10 0xbbba59 in SSLNetVConnection::net_read_io(NetHandler*, EThread*)
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:489
> #11 0xbda09c in NetHandler::mainNetEvent(int, Event*)
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513
> #12 0xc85089 in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #13 0xc85089 in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #14 0xc85089 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268
> #15 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826
> #16 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> Thread T2 ([ET_NET 1]) created by T0 ([ET_NET 0]) here:
> #0 0x2aaf08be286a in __interceptor_pthread_create
> ../../.././libsanitizer/asan/asan_interceptors.cc:183
> #1 0xc81aa5 in ink_thread_create ../../lib/ts/ink_thread.h:148
> #2 0xc81aa5 in Thread::start(char const*, unsigned long, void*
> (*)(void*), void*)
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:103
> #3 0xc8a026 in EventProcessor::start(int, unsigned long)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEventProcessor.cc:140
> #4 0x498d0b in main /usr/local/src/trafficserver/proxy/Main.cc:1636
> #5 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> SUMMARY: AddressSanitizer: heap-use-after-free
> ../iocore/eventsystem/I_Continuation.h:146 Continuation::handleEvent(int,
> void*)
> Shadow bytes around the buggy address:
> 0x0c307fff98c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
> 0x0c307fff9900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> =>0x0c307fff9910: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9960: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Heap right redzone: fb
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack partial redzone: f4
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Contiguous container OOB:fc
> ASan internal: fe
> ==31101==ABORTING
> traffic_server: using root directory '/opt/ats'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
