shinrich edited a comment on pull request #7845:
URL: https://github.com/apache/trafficserver/pull/7845#issuecomment-871728314
I pulled in this change and the other H2 cleanups into our 9.1 branch and
tried it in production. It crashed pretty quickly with the following stack.
Looks like we are having problems on shutdown. The HttpSM (the cont in the
read_vio) has already been deleted and its mutex has been cleared.
```
(gdb) bt
#0 Mutex_lock (t=0x2aebf5373c80, m=0x0) at
/home/shinrich/build-new/_build/build_release_posix-x86_64_gcc_8/trafficserver9.1/build/../../../../_scm/trafficserver9.1/iocore/eventsystem/I_Lock.h:439
#1 Mutex_lock (t=0x2aebf5373c80, m=...) at
/home/shinrich/build-new/_build/build_release_posix-x86_64_gcc_8/trafficserver9.1/build/../../../../_scm/trafficserver9.1/iocore/eventsystem/I_Lock.h:351
#2 MutexLock::MutexLock (this=0x2aebfbf094e0, am=..., t=0x2aebf5373c80)
at
/home/shinrich/build-new/_build/build_release_posix-x86_64_gcc_8/trafficserver9.1/build/../../../../_scm/trafficserver9.1/iocore/eventsystem/I_Lock.h:439
#3 0x00000000006333d5 in Http2Stream::main_event_handler(int, void*) ()
at
/home/shinrich/build-new/_build/build_release_posix-x86_64_gcc_8/trafficserver9.1/build/../../../../_scm/trafficserver9.1/include/tscore/ink_thread.h:124
#4 0x00000000007db23d in handleEvent (data=0x2aec33e039e0, event=104,
this=0x2aee507b8e00) at
../../../../../../_scm/trafficserver9.1/iocore/eventsystem/I_Continuation.h:219
#5 handleEvent (data=0x2aec33e039e0, event=104, this=0x2aee507b8e00) at
../../../../../../_scm/trafficserver9.1/iocore/eventsystem/I_Continuation.h:215
#6 EThread::process_event(Event*, int) () at
../../../../../../_scm/trafficserver9.1/iocore/eventsystem/UnixEThread.cc:164
#7 0x00000000007db80e in EThread::process_queue
(this=this@entry=0x2aebf5373c80,
NegativeQueue=NegativeQueue@entry=0x2aebfbf09600,
ev_count=ev_count@entry=0x2aebfbf095fc,
nq_count=nq_count@entry=0x2aebfbf095f8) at
../../../../../../_scm/trafficserver9.1/iocore/eventsystem/UnixEThread.cc:199
#8 0x00000000007dbc74 in EThread::execute_regular
(this=this@entry=0x2aebf5373c80) at
../../../../../../_scm/trafficserver9.1/iocore/eventsystem/UnixEThread.cc:259
#9 0x00000000007dc2e2 in execute (this=0x2aebf5373c80) at
../../../../../../_scm/trafficserver9.1/iocore/eventsystem/UnixEThread.cc:364
#10 EThread::execute (this=0x2aebf5373c80) at
../../../../../../_scm/trafficserver9.1/iocore/eventsystem/UnixEThread.cc:342
#11 0x00000000007da92a in spawn_thread_internal (a=0x2aebefa863c0) at
../../../../../../_scm/trafficserver9.1/iocore/eventsystem/Thread.cc:92
#12 0x00002aebede9dea5 in start_thread (arg=0x2aebfbf0b700) at
pthread_create.c:307
#13 0x00002aebeebd49fd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
```
The Http2Stream _history is
```
{history = {{location = {file = 0x812320
"../../../../../../_scm/trafficserver9.1/proxy/http2/Http2Stream.cc",
func = 0x8126c0 <Http2Stream::send_tracked_event(Event*, int,
VIO*)::__FUNCTION__> "send_tracked_event", line = 581}, event = 101, reentrancy
= 0}, {location = {
file = 0x812320
"../../../../../../_scm/trafficserver9.1/proxy/http2/Http2Stream.cc", func =
0x812740 <Http2Stream::main_event_handler(int, void*)::__FUNCTION__>
"main_event_handler", line = 141},
event = 101, reentrancy = 0}, {location = {file = 0x812320
"../../../../../../_scm/trafficserver9.1/proxy/http2/Http2Stream.cc",
func = 0x8126e0 <Http2Stream::initiating_close()::__FUNCTION__>
"initiating_close", line = 515}, event = 34463, reentrancy = 0}, {location = {
file = 0x812320
"../../../../../../_scm/trafficserver9.1/proxy/http2/Http2Stream.cc", func =
0x8126c0 <Http2Stream::send_tracked_event(Event*, int, VIO*)::__FUNCTION__>
"send_tracked_event",
line = 581}, event = 104, reentrancy = 0}, {location = {file =
0x812320 "../../../../../../_scm/trafficserver9.1/proxy/http2/Http2Stream.cc",
func = 0x812740 <Http2Stream::main_event_handler(int,
void*)::__FUNCTION__> "main_event_handler", line = 141}, event = 104,
reentrancy = 0}, {location = {file = 0x0, func = 0x0, line = 0}, event = 0,
reentrancy = 0} <repeats 60 times>}, history_pos = 5}
```
It looks like the HttpSM history is also still around. The end of that is
```
{location = {file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc",
func = 0x7fe6c0 <HttpSM::setup_server_transfer()::__FUNCTION__>
"setup_server_transfer", line = 6838}, event = 34463, reentrancy = 11},
{location = {
file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7fed20
<HttpSM::tunnel_handler_ua(int, HttpTunnelConsumer*)::__FUNCTION__>
"tunnel_handler_ua", line = 3345},
event = 104, reentrancy = 0}, {location = {file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc",
func = 0x7fedc0 <HttpSM::tunnel_handler(int, void*)::__FUNCTION__>
"tunnel_handler", line = 3015}, event = 2301, reentrancy = 1}, {location = {
file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7ff090
<HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback",
line = 1445}, event = 60000,
reentrancy = 2}, {location = {file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7ff070
<HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout",
line = 1485}, event = 60000, reentrancy = 2}, {location = {file =
0x7fad38 "../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc",
func = 0x7ff090 <HttpSM::state_api_callback(int,
void*)::__FUNCTION__> "state_api_callback", line = 1445}, event = 60000,
reentrancy = 3}, {location = {
file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7ff070
<HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line
= 1485}, event = 60000,
reentrancy = 3}, {location = {file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7ff090
<HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback",
line = 1445}, event = 60000, reentrancy = 4}, {location = {file =
0x7fad38 "../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc",
func = 0x7ff070 <HttpSM::state_api_callout(int,
void*)::__FUNCTION__> "state_api_callout", line = 1485}, event = 60000,
reentrancy = 4}, {location = {
file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7ff090
<HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback",
line = 1445}, event = 60000,
reentrancy = 5}, {location = {file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7ff070
<HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout",
line = 1485}, event = 60000, reentrancy = 5}, {location = {file =
0x7fad38 "../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc",
---Type <return> to continue, or q <return> to quit---
func = 0x7ff090 <HttpSM::state_api_callback(int,
void*)::__FUNCTION__> "state_api_callback", line = 1445}, event = 60000,
reentrancy = 6}, {location = {
file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7ff070
<HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line
= 1485}, event = 60000,
reentrancy = 6}, {location = {file = 0x7fad38
"../../../../../../_scm/trafficserver9.1/proxy/http/HttpSM.cc", func = 0x7fe668
<HttpSM::kill_this()::__FUNCTION__> "kill_this", line = 7157}, event = 34463,
reentrancy = 0}
```
It seems that the Http2Stream had already delivered an EOS which caused the
HttpSM to clean itself up.
Will continue to poke around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
