[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-10-06 Thread Susan Hinrichs (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945958#comment-14945958
 ] 

Susan Hinrichs commented on TS-3710:


Typoed the bug number in the commit comment.  This commit belongs with this 
issue.

Commit 1859562086b330eed6eda637f5f98a3431db5915 in trafficserver's branch 
refs/heads/master from shinrich
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1859562 ]
TS-3701 - Crash in trampoline cleanup

> Crash in TLS with 6.0.0, related to the session cleanup additions
> -
>
> Key: TS-3710
> URL: https://issues.apache.org/jira/browse/TS-3710
> Project: Traffic Server
>  Issue Type: Bug
>  Components: SSL
>Affects Versions: 5.3.0
>Reporter: Leif Hedstrom
>Assignee: Susan Hinrichs
>Priority: Critical
>  Labels: yahoo
> Fix For: 6.1.0
>
> Attachments: ts-3710-2.diff, ts-3710-8-26-15.diff, 
> ts-3710-final-2.diff, ts-3710.diff
>
>
> {code}
> ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
> 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
> READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
> #0 0xb9f968 in Continuation::handleEvent(int, void*) 
> ../../iocore/eventsystem/I_Continuation.h:145
> #1 0xb9f968 in read_signal_and_update 
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
> #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
> #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
> ../../iocore/eventsystem/I_Continuation.h:145
> #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
> #5 0xc21ffe in Continuation::handleEvent(int, void*) 
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
> #6 0xc21ffe in EThread::process_event(Event*, int) 
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
> #7 0xc241f7 in EThread::execute() 
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
> #8 0xc20c18 in spawn_thread_internal 
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
> #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
> #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
> 0x60649f48 is located 8 bytes inside of 56-byte region 
> [0x60649f40,0x60649f78)
> freed by thread T8 ([ET_NET 7]) here:
> #0 0x2b8db1bf3117 in operator delete(void*) 
> ../../.././libsanitizer/asan/asan_new_delete.cc:81
> #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
> /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
> #2 0xbb2eef in Continuation::handleEvent(int, void*) 
> ../../iocore/eventsystem/I_Continuation.h:145
> #3 0xbb2eef in read_signal_and_update 
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
> #4 0xbb2eef in read_signal_done 
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
> #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
> /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
> #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
> /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
> #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
> /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
> #8 0xc24e89 in Continuation::handleEvent(int, void*) 
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
> #9 0xc24e89 in EThread::process_event(Event*, int) 
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
> #10 0xc24e89 in EThread::execute() 
> /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
> #11 0xc20c18 in spawn_thread_internal 
> /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
> #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
> previously allocated by thread T8 ([ET_NET 7]) here:
> #0 0x2b8db1bf2c9f in operator new(unsigned long) 
> ../../.././libsanitizer/asan/asan_new_delete.cc:50
> #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
> /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
> #2 0xb888e9 in Continuation::handleEvent(int, void*) 
> ../../iocore/eventsystem/I_Continuation.h:145
> #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
> /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
> #4 0xc24e89 in Continuation::handleEvent(int, void*) 
> /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
> #5 0xc24e89 in EThread::process_event(Event*, int) 
>

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-20 Thread Bryan Call (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633500#comment-14633500
 ] 

Bryan Call commented on TS-3710:


Adding the NULL check makes it crash less, but as [~amc] stated you can't 
guarantee that it is always NULL.

I am running an ASAN build in production to see if I can track it down.

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Affects Versions: 5.3.0
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
  Labels: yahoo
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
 #4 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #5 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #6 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #7 0xc20c18 in spawn_thread_internal

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-20 Thread Susan Hinrichs (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633650#comment-14633650
 ] 

Susan Hinrichs commented on TS-3710:


I'd definitely try the do_io_read(NULL, 0 , NULL) in ioCompleteEvent before the 
send_plugin_event calls.  That should clear the read.vio._cont before the 
trampoline is deleted.  Based on the cores I looked at over the weekend, it 
definitely looked like the problem continuation was the 
SSLNextProtocolTrampoline.

 Although that was the last patch that [~zwoop] tried.  While it might have 
slowed down the problem.  It did not stop it completely.

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Affects Versions: 5.3.0
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
  Labels: yahoo
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
 #4 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #5 0xc24e89 in

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-19 Thread Leif Hedstrom (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632765#comment-14632765
 ] 

Leif Hedstrom commented on TS-3710:
---

If you want to, we can restore the patch that was reverted due to this. With 
that patch active, it was crashing very, very frequently (10+ times / day on a 
low traffic server). As @amc has pointed out before, it's possible that was a 
red herring, and only made an pre-existing condition worse.

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Affects Versions: 5.3.0
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
  Labels: yahoo
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
 #4 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #5 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #6 0xc24e89 in EThread::execute()

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-18 Thread Sudheer Vinukonda (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632457#comment-14632457
 ] 

Sudheer Vinukonda commented on TS-3710:
---

AFAICT, this to me seems like regressions caused since 5.0.x. I don't recall 
seeing this many core dumps in this area until early 5.0.x (middle/late 2014). 
It seems to me that recently, there have been a bunch of NULL *do_io_read* 
being added all over the place, without properly understanding the 
impact/consequences. I'd actually suggest to re-review all such commits and 
revert the ones that could cause more harm than help. Otherwise, I fear, we'd 
end up adding band-aid fixes in a never-ending loop. 

Just my 2c.

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Affects Versions: 5.3.0
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
  Labels: yahoo
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
 #4 0xc24e89 in Continuation::handleEvent(int, void*)

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-18 Thread Alan M. Carroll (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632441#comment-14632441
 ] 

Alan M. Carroll commented on TS-3710:
-

It is seriously wrong to have a continuation with a {{NULL}} handler. That 
makes me suspect it's a deleted / freed continuation and given the call stacks 
is likely the SSLNextProtocolTrampoline which deletes itself after the 
{{ioCompletionEvent}} invocation. As [~shinrich] noted we had similar crashes 
for the non-TLS accept handling which seem to have been fixed by nulling out 
the VC read VIO continuation by calling {{do_io_read(NULL, 0, NULL}} (see 
{{ProtocolProbeTrampoline::ioCompletionEvent}}. I recommend trying that in 
preference to checking the handler - in that case the {{NULL}} check already 
there should work. It's unclear to me that if this is a dead continuation you 
can count on the handler being {{NULL}} in all cases.

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Affects Versions: 5.3.0
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
  Labels: yahoo
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*)

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-18 Thread Bryan Call (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632419#comment-14632419
 ] 

Bryan Call commented on TS-3710:


Seeing the original core in the description on 5.3.x in production.  It happens 
when there is an accept timeout and read.vio.op == VIO::READ.  There is no read 
continuation handler and it cores when trying to call a NULL function pointer.

Going to test with this patch in production:
{code}
diff --git a/iocore/net/UnixNetVConnection.cc b/iocore/net/UnixNetVConnection.cc
index f32a25f..b831102 100644
--- a/iocore/net/UnixNetVConnection.cc
+++ b/iocore/net/UnixNetVConnection.cc
@@ -141,7 +141,7 @@ static inline int
 read_signal_and_update(int event, UnixNetVConnection *vc)
 {
   vc-recursion++;
-  if (vc-read.vio._cont) {
+  if (vc-read.vio._cont  vc-handler) {
 vc-read.vio._cont-handleEvent(event, vc-read.vio);
   } else {
 switch (event) {
{code}

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Affects Versions: 5.3.0
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
  Labels: yahoo
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*)

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-09 Thread Bryan Call (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620690#comment-14620690
 ] 

Bryan Call commented on TS-3710:


Reverted c8c9c4fa0dad3 on 6.0.x

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
 #4 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #5 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #6 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #7 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #8 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 Thread T8 ([ET_NET 7]) created by T0 ([ET_NET 0]) here:
 #0 0x2b8db1bc186a in __interceptor_pthread_create

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-06 Thread Leif Hedstrom (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615088#comment-14615088
 ] 

Leif Hedstrom commented on TS-3710:
---

I backed out those changes as well, and I'm running again without commit 
c8c9c4fa0dad3, and so far no crashes and no hangs. I will leave it like this 
for a fe days, to make sure the hanging we were seeing is not from something 
else unrelated to this.

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
 #4 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #5 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #6 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #7 0xc20c18 in spawn_thread_internal

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-06 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615068#comment-14615068
 ] 

ASF subversion and git services commented on TS-3710:
-

Commit 5defa51cd3ddcce05b0be4aa3f7b9d5a6f87a111 in trafficserver's branch 
refs/heads/master from shinrich
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=5defa51 ]

Revert TS-3710: ASAN crash in TLS with 6.0.0

This reverts commit f71d0685333d7853a7ca5ea98fc43b18de1f5488.

Still seeing issues.  Backing out this change until we get the whole issue 
figured out.


 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710-final-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
 #4 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #5 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #6

[jira] [Commented] (TS-3710) Crash in TLS with 6.0.0, related to the session cleanup additions

2015-07-04 Thread bettydramit (JIRA) 

[ 
https://issues.apache.org/jira/browse/TS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614101#comment-14614101
 ] 

bettydramit commented on TS-3710:
-

It can be backport to 5.2.x and 5.3.x ?

 Crash in TLS with 6.0.0, related to the session cleanup additions
 -

 Key: TS-3710
 URL: https://issues.apache.org/jira/browse/TS-3710
 Project: Traffic Server
  Issue Type: Bug
  Components: SSL
Reporter: Leif Hedstrom
Assignee: Susan Hinrichs
Priority: Critical
 Fix For: 6.1.0

 Attachments: ts-3710-2.diff, ts-3710.diff


 {code}
 ==9570==ERROR: AddressSanitizer: heap-use-after-free on address 
 0x60649f48 at pc 0xb9f969 bp 0x2b8dbc348920 sp 0x2b8dbc348918
 READ of size 8 at 0x60649f48 thread T8 ([ET_NET 7])
 #0 0xb9f968 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #1 0xb9f968 in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #2 0xb9f968 in UnixNetVConnection::mainEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1115
 #3 0xb7daf7 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #4 0xb7daf7 in InactivityCop::check_inactivity(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:102
 #5 0xc21ffe in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #6 0xc21ffe in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #7 0xc241f7 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:207
 #8 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #9 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 #10 0x2b8db585f1ac in __clone (/lib64/libc.so.6+0xf61ac)
 0x60649f48 is located 8 bytes inside of 56-byte region 
 [0x60649f40,0x60649f78)
 freed by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf3117 in operator delete(void*) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:81
 #1 0xb5b20e in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:89
 #2 0xbb2eef in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xbb2eef in read_signal_and_update 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:142
 #4 0xbb2eef in read_signal_done 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:203
 #5 0xbb2eef in UnixNetVConnection::readSignalDone(int, NetHandler*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:957
 #6 0xb55d6d in SSLNetVConnection::net_read_io(NetHandler*, EThread*) 
 /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:480
 #7 0xb748fc in NetHandler::mainNetEvent(int, Event*) 
 /usr/local/src/trafficserver/iocore/net/UnixNet.cc:516
 #8 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #9 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #10 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #11 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #12 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 previously allocated by thread T8 ([ET_NET 7]) here:
 #0 0x2b8db1bf2c9f in operator new(unsigned long) 
 ../../.././libsanitizer/asan/asan_new_delete.cc:50
 #1 0xb59f8b in SSLNextProtocolAccept::mainEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:134
 #2 0xb888e9 in Continuation::handleEvent(int, void*) 
 ../../iocore/eventsystem/I_Continuation.h:145
 #3 0xb888e9 in NetAccept::acceptFastEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/net/UnixNetAccept.cc:466
 #4 0xc24e89 in Continuation::handleEvent(int, void*) 
 /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:145
 #5 0xc24e89 in EThread::process_event(Event*, int) 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:128
 #6 0xc24e89 in EThread::execute() 
 /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:252
 #7 0xc20c18 in spawn_thread_internal 
 /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:85
 #8 0x2b8db3ff6df4 in start_thread (/lib64/libpthread.so.0+0x7df4)
 Thread T8 ([ET_NET 7]) created by T0 ([ET_NET 0]) here:
 #0 0x2b8db1bc186a in __interceptor_pthread_create