[
https://issues.apache.org/jira/browse/TS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Susan Hinrichs updated TS-3405:
-------------------------------
Attachment: fetchsm-change.diff
Still no crashes. I applied the fetchsm change I was playing with yesterday
(fetchsm-change.diff), and I no longer see the CPU spin.
In process_fetch_read, the original would call the stream version of
InvokePluginExt and return immediately never calling cleanup even if the
incoming event was EOS. So the closed socket with stick around continually
signaling EOS. I changed the logic to call cleanup after InvokePluginExt if
the event was EOS.
[~briang] can you verify that this is the correct thing to do and will not
negatively impact SPDY assumptions?
In my testing I was running with both of Ryo's fix-h2 patches and the
fetchsm-change patch.
> Memory use after free in HTTP/2
> -------------------------------
>
> Key: TS-3405
> URL: https://issues.apache.org/jira/browse/TS-3405
> Project: Traffic Server
> Issue Type: Bug
> Components: HTTP/2
> Reporter: Bryan Call
> Fix For: 5.3.0
>
> Attachments: 0002-fix-h2.patch, fetchsm-change.diff, fix-h2.patch
>
>
> From Leif running on docs.trafficserver.apache.org:
>
> {code}
> traffic_server: using root directory '/opt/ats'
> =================================================================
> ==31101==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61800000c888 at pc 0x4f3558 bp 0x2aaf10c88930 sp 0x2aaf10c88928
> READ of size 8 at 0x61800000c888 thread T2 ([ET_NET 1])
> #0 0x4f3557 in Continuation::handleEvent(int, void*)
> ../iocore/eventsystem/I_Continuation.h:146
> #1 0x4f3557 in FetchSM::InvokePluginExt(int)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:301
> #2 0x4f3a7a in FetchSM::process_fetch_read(int)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:465
> #3 0x4f5112 in FetchSM::fetch_handler(int, void*)
> /usr/local/src/trafficserver/proxy/FetchSM.cc:514
> #4 0x59f1b7 in Continuation::handleEvent(int, void*)
> ../iocore/eventsystem/I_Continuation.h:146
> #5 0x59f1b7 in PluginVC::process_read_side(bool)
> /usr/local/src/trafficserver/proxy/PluginVC.cc:640
> #6 0x5abcb9 in PluginVC::main_handler(int, void*)
> /usr/local/src/trafficserver/proxy/PluginVC.cc:206
> #7 0xc821fe in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #8 0xc821fe in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #9 0xc84819 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:238
> #10 0xc80e18 in spawn_thread_internal
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:88
> #11 0x2aaf0b083df2 in start_thread (/lib64/libpthread.so.0+0x7df2)
> #12 0x2aaf0c8ec1ac in clone (/lib64/libc.so.6+0xf61ac)
> 0x61800000c888 is located 8 bytes inside of 816-byte region
> [0x61800000c880,0x61800000cbb0)
> freed by thread T0 ([ET_NET 0]) here:
> #0 0x2aaf08c131c7 in __interceptor_free
> ../../.././libsanitizer/asan/asan_malloc_linux.cc:62
> #1 0x7b7d42 in Http2ClientSession::do_io_close(int)
> /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:194
> #2 0x7b7d42 in Http2ClientSession::main_event_handler(int, void*)
> /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:237
> #3 0xc1351f in Continuation::handleEvent(int, void*)
> ../../iocore/eventsystem/I_Continuation.h:146
> #4 0xc1351f in read_signal_and_update
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140
> #5 0xc1351f in read_signal_done
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185
> #6 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*)
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939
> #7 0xbbabf8 in SSLNetVConnection::net_read_io(NetHandler*, EThread*)
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:596
> #8 0xbda09c in NetHandler::mainNetEvent(int, Event*)
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513
> #9 0xc85089 in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #10 0xc85089 in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #11 0xc85089 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268
> #12 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826
> #13 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> previously allocated by thread T0 ([ET_NET 0]) here:
> #0 0x2aaf08c1393b in __interceptor_posix_memalign
> ../../.././libsanitizer/asan/asan_malloc_linux.cc:130
> #1 0x2aaf09afd2f9 in ats_memalign
> /usr/local/src/trafficserver/lib/ts/ink_memory.cc:96
> #2 0x7cd804 in ClassAllocator<Http2ClientSession>::alloc()
> ../../lib/ts/Allocator.h:124
> #3 0x7cd804 in Http2SessionAccept::accept(NetVConnection*, MIOBuffer*,
> IOBufferReader*)
> /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:57
> #4 0x7cd3c4 in Http2SessionAccept::mainEvent(int, void*)
> /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:69
> #5 0xbc2fae in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*)
> /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:101
> #6 0xc1351f in Continuation::handleEvent(int, void*)
> ../../iocore/eventsystem/I_Continuation.h:146
> #7 0xc1351f in read_signal_and_update
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140
> #8 0xc1351f in read_signal_done
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185
> #9 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*)
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939
> #10 0xbbba59 in SSLNetVConnection::net_read_io(NetHandler*, EThread*)
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:489
> #11 0xbda09c in NetHandler::mainNetEvent(int, Event*)
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513
> #12 0xc85089 in Continuation::handleEvent(int, void*)
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146
> #13 0xc85089 in EThread::process_event(Event*, int)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144
> #14 0xc85089 in EThread::execute()
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268
> #15 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826
> #16 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> Thread T2 ([ET_NET 1]) created by T0 ([ET_NET 0]) here:
> #0 0x2aaf08be286a in __interceptor_pthread_create
> ../../.././libsanitizer/asan/asan_interceptors.cc:183
> #1 0xc81aa5 in ink_thread_create ../../lib/ts/ink_thread.h:148
> #2 0xc81aa5 in Thread::start(char const*, unsigned long, void*
> (*)(void*), void*)
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:103
> #3 0xc8a026 in EventProcessor::start(int, unsigned long)
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEventProcessor.cc:140
> #4 0x498d0b in main /usr/local/src/trafficserver/proxy/Main.cc:1636
> #5 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4)
> SUMMARY: AddressSanitizer: heap-use-after-free
> ../iocore/eventsystem/I_Continuation.h:146 Continuation::handleEvent(int,
> void*)
> Shadow bytes around the buggy address:
> 0x0c307fff98c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff98f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
> 0x0c307fff9900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> =>0x0c307fff9910: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c307fff9960: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Heap right redzone: fb
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack partial redzone: f4
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Contiguous container OOB:fc
> ASan internal: fe
> ==31101==ABORTING
> traffic_server: using root directory '/opt/ats'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
