[
https://issues.apache.org/jira/browse/TS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335574#comment-14335574
]
Susan Hinrichs commented on TS-3405:
------------------------------------
I reverted to what is currently on master. After a few page views, ATS gets
into a state where it is consistently using more than 100% CPU. I think what
is happening is that the origin server is closing the connection (we get a
TS_EVENT_VCONN_EOS in FetchSM). FetchSM sends a BODY_DONE to
Http2ClientSession but the http_vc (origin socket?) is never closed, so with
each poll interval the EOS is reported again and again.
I tried an adjustment in FetchSM::process_fetch_read() to call cleanup() in the
EOS case, but now I get a segv. Fetchsm is bogus but a Http2ClientSession
still refers to it.
[~briang] what was SPDY's approach to handling origin server initiated
shutdown? Did it report the EOS to the SpdyClient class and rely on it to
close up the appropriate VCs? Or did FetchSM close up the origin server state
itself?
> Memory use after free in HTTP/2
> -------------------------------
>
> Key: TS-3405
> URL: https://issues.apache.org/jira/browse/TS-3405
> Project: Traffic Server
> Issue Type: Bug
> Components: HTTP/2
> Reporter: Bryan Call
> Fix For: 5.3.0
>
> Attachments: fix-h2.patch
>
>
> From Leif running on docs.trafficserver.apache.org:
>
> {code}
> traffic_server: using root directory '/opt/ats'
> =================================================================
> ==31101==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61800000c888 at pc 0x4f3558 bp 0x2aaf10c88930 sp 0x2aaf10c88928
> READ of size 8 at 0x61800000c888 thread T2 ([ET_NET 1])
> #0 0x4f3557 in Continuation::handleEvent(int, void*)
> ../iocore/eventsystem/I_Continuation.h:146
> #1 0x4f3557 in FetchSM::InvokePluginExt(int)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:301
> #2 0x4f3a7a in FetchSM::process_fetch_read(int)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:465
> #3 0x4f5112 in FetchSM::fetch_handler(int, void*)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:514
> #4 0x59f1b7 in Continuation::handleEvent(int, void*)
> ../iocore/eventsystem/I_Continuation.h:146
> #5 0x59f1b7 in PluginVC::process_read_side(bool)
> /usr/local/src/trafficserver/proxy/PluginVC.cc:640
> #6 0x5abcb9 in PluginVC::main_handler(int, void*)
> /usr/local/src/trafficserver/proxy/PluginVC.cc:206
> #7 0xc821fe in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #8 0xc821fe in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #9 0xc84819 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:238
> #10 0xc80e18 in spawn_thread_internal
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:88
> #11 0x2aaf0b083df2 in start_thread (/lib64/libpthread.so.0+0x7df2)
> #12 0x2aaf0c8ec1ac in clone (/lib64/libc.so.6+0xf61ac)
> 0x61800000c888 is located 8 bytes inside of 816-byte region
> [0x61800000c880,0x61800000cbb0)
> freed by thread T0 ([ET_NET 0]) here:
> #0 0x2aaf08c131c7 in __interceptor_free
> ../../.././libsanitizer/asan/asan_malloc_linux.cc:62
> #1 0x7b7d42 in Http2ClientSession::do_io_close(int)
> /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:194
> #2 0x7b7d42 in Http2ClientSession::main_event_handler(int, void*)
> /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:237
> #3 0xc1351f in Continuation::handleEvent(int, void*)
> ../../iocore/eventsystem/I_Continuation.h:146
> #4 0xc1351f in read_signal_and_update
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140
> #5 0xc1351f in read_signal_done
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185
> #6 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*)
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939
> #7 0xbbabf8 in SSLNetVConnection::net_read_io(NetHandler*, EThread*)
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:596
> #8 0xbda09c in NetHandler::mainNetEvent(int, Event*)
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513
> #9 0xc85089 in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #10 0xc85089 in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #11 0xc85089 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268
> #12 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826
> #13 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> previously allocated by thread T0 ([ET_NET 0]) here:
> #0 0x2aaf08c1393b in __interceptor_posix_memalign
> ../../.././libsanitizer/asan/asan_malloc_linux.cc:130
> #1 0x2aaf09afd2f9 in ats_memalign
> /usr/local/src/trafficserver/lib/ts/ink_memory.cc:96
> #2 0x7cd804 in ClassAllocator<Http2ClientSession>::alloc()
> ../../lib/ts/Allocator.h:124
> #3 0x7cd804 in Http2SessionAccept::accept(NetVConnection*, MIOBuffer*,
> IOBufferReader*)
> /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:57
> #4 0x7cd3c4 in Http2SessionAccept::mainEvent(int, void*)
> /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:69
> #5 0xbc2fae in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*)
> /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:101
> #6 0xc1351f in Continuation::handleEvent(int, void*)
> ../../iocore/eventsystem/I_Continuation.h:146
> #7 0xc1351f in read_signal_and_update
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140
> #8 0xc1351f in read_signal_done
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185
> #9 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*)
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939
> #10 0xbbba59 in SSLNetVConnection::net_read_io(NetHandler*, EThread*)
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:489
> #11 0xbda09c in NetHandler::mainNetEvent(int, Event*)
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513
> #12 0xc85089 in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #13 0xc85089 in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #14 0xc85089 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268
> #15 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826
> #16 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> Thread T2 ([ET_NET 1]) created by T0 ([ET_NET 0]) here:
> #0 0x2aaf08be286a in __interceptor_pthread_create
> ../../.././libsanitizer/asan/asan_interceptors.cc:183
> #1 0xc81aa5 in ink_thread_create ../../lib/ts/ink_thread.h:148
> #2 0xc81aa5 in Thread::start(char const*, unsigned long, void*
> (*)(void*), void*)
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:103
> #3 0xc8a026 in EventProcessor::start(int, unsigned long)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEventProcessor.cc:140
> #4 0x498d0b in main /usr/local/src/trafficserver/proxy/Main.cc:1636
> #5 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> SUMMARY: AddressSanitizer: heap-use-after-free
> ../iocore/eventsystem/I_Continuation.h:146 Continuation::handleEvent(int,
> void*)
> Shadow bytes around the buggy address:
> 0x0c307fff98c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
> 0x0c307fff9900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> =>0x0c307fff9910: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9960: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Heap right redzone: fb
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack partial redzone: f4
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Contiguous container OOB:fc
> ASan internal: fe
> ==31101==ABORTING
> traffic_server: using root directory '/opt/ats'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
