[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=31221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-31221
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 27/Oct/16 03:21
Start Date: 27/Oct/16 03:21
Worklog Time Spent: 10m 
  Work Description: Github user PSUdaemon closed the pull request at:

https://github.com/apache/trafficserver/pull/1157


Issue Time Tracking
---

Worklog Id: (was: 31221)
Time Spent: 8h  (was: 7h 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.0.0
>
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840
> --- count=2 ---
> id=19
> this=0x2ae5b6bbec00
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=31220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-31220
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 27/Oct/16 02:40
Start Date: 27/Oct/16 02:40
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1157
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/1109/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 31220)
Time Spent: 7h 50m  (was: 7h 40m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.0.0
>
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=31219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-31219
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 27/Oct/16 02:36
Start Date: 27/Oct/16 02:36
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1157
  
Linux build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/1002/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 31219)
Time Spent: 7h 40m  (was: 7.5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.0.0
>
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=31218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-31218
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 27/Oct/16 02:26
Start Date: 27/Oct/16 02:26
Worklog Time Spent: 10m 
  Work Description: GitHub user gtenev opened a pull request:

https://github.com/apache/trafficserver/pull/1157

TS-4916 Add safety net to avoid H2-infinite-loop deadlock.

Current Http2ConnectionState implementation uses a memory pool for
instantiating streams and DLL<> stream_list for storing active streams.
Destroying a stream before deleting it from stream_list and then creating
a new one + reusing the same chunk from the memory pool right away always
leads to destroying the DLL structure (deadlocks, inconsistencies).
Added a safety net since the consequences are disastrous.
Until the design/implementation changes it seems less error prone
to (double) delete before destroying (noop if already deleted).

(cherry picked from commit a6f9337f61c980739e08bbefeb5f6409b7dcdac1)

Conflicts:
proxy/http2/Http2ConnectionState.cc

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gtenev/trafficserver h2_loop

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/trafficserver/pull/1157.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1157


commit 8b3d67d8f3bdb7e837c766e57ae75286f1eb01a3
Author: Gancho Tenev 
Date:   2016-10-17T22:10:12Z

TS-4916 Add safety net to avoid H2-infinite-loop deadlock.

Current Http2ConnectionState implementation uses a memory pool for
instantiating streams and DLL<> stream_list for storing active streams.
Destroying a stream before deleting it from stream_list and then creating
a new one + reusing the same chunk from the memory pool right away always
leads to destroying the DLL structure (deadlocks, inconsistencies).
Added a safety net since the consequences are disastrous.
Until the design/implementation changes it seems less error prone
to (double) delete before destroying (noop if already deleted).

(cherry picked from commit a6f9337f61c980739e08bbefeb5f6409b7dcdac1)

Conflicts:
proxy/http2/Http2ConnectionState.cc




Issue Time Tracking
---

Worklog Id: (was: 31218)
Time Spent: 7.5h  (was: 7h 20m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.0.0
>
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30875
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 19/Oct/16 15:04
Start Date: 19/Oct/16 15:04
Worklog Time Spent: 10m 
  Work Description: Github user zwoop closed the pull request at:

https://github.com/apache/trafficserver/pull/1117


Issue Time Tracking
---

Worklog Id: (was: 30875)
Time Spent: 7h 20m  (was: 7h 10m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840
> --- count=2 ---
> id=19
> this=0x2ae5b6bbec00
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30854
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 21:14
Start Date: 18/Oct/16 21:14
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
Linux build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/942/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30854)
Time Spent: 7h 10m  (was: 7h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30853
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 21:11
Start Date: 18/Oct/16 21:11
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/1050/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30853)
Time Spent: 7h  (was: 6h 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30852
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 21:09
Start Date: 18/Oct/16 21:09
Worklog Time Spent: 10m 
  Work Description: Github user shinrich commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
Looks good to me as well.


Issue Time Tracking
---

Worklog Id: (was: 30852)
Time Spent: 6h 50m  (was: 6h 40m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840
> --- count=2 ---
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30834
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 19:49
Start Date: 18/Oct/16 19:49
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
Linux build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/940/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30834)
Time Spent: 6h 40m  (was: 6.5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30833
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 19:46
Start Date: 18/Oct/16 19:46
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/1048/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30833)
Time Spent: 6.5h  (was: 6h 20m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30832=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30832
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 19:35
Start Date: 18/Oct/16 19:35
Worklog Time Spent: 10m 
  Work Description: Github user zwoop commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
Sigh, it failed again on the bison generated files :-/. [approve ci].


Issue Time Tracking
---

Worklog Id: (was: 30832)
Time Spent: 6h 20m  (was: 6h 10m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30820
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 17:38
Start Date: 18/Oct/16 17:38
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/1041/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30820)
Time Spent: 6h 10m  (was: 6h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30818
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 17:34
Start Date: 18/Oct/16 17:34
Worklog Time Spent: 10m 
  Work Description: Github user gtenev commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Chatted with @shinrich offline and she is going to mark related fixes 
TS-4813 and TS-4507 for backport from 7.0.0 to 6.2.1, which should take care of 
the "missing to delete the stream cases". 

Submitted a new PR against master #1117 to add the "safety net" delete 
stream call in `destroy()` before `THREAD_FREE()` and backport to 6.2.1 
(changes are not 6.2.1 specific anymore).


Issue Time Tracking
---

Worklog Id: (was: 30818)
Time Spent: 5h 50m  (was: 5h 40m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30819
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 17:34
Start Date: 18/Oct/16 17:34
Worklog Time Spent: 10m 
  Work Description: Github user gtenev closed the pull request at:

https://github.com/apache/trafficserver/pull/1100


Issue Time Tracking
---

Worklog Id: (was: 30819)
Time Spent: 6h  (was: 5h 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840
> --- count=2 ---
> id=19
> this=0x2ae5b6bbec00
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30817=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30817
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 17:34
Start Date: 18/Oct/16 17:34
Worklog Time Spent: 10m 
  Work Description: Github user zwoop commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1117#discussion_r83908521
  
--- Diff: proxy/http2/Http2Stream.cc ---
@@ -675,6 +675,17 @@ Http2Stream::destroy()
   }
   chunked_handler.clear();
   super::destroy();
+
+  // Current Http2ConnectionState implementation uses a memory pool for 
instantiating streams and DLL<> stream_list for storing
+  // active streams. Destroying a stream before deleting it from 
stream_list and then creating a new one + reusing the same chunk
+  // from the memory pool right away always leads to destroying the DLL 
structure (deadlocks, inconsistencies).
+  // The following is meant as a safety net since the consequences are 
disastrous. Until the design/implementation changes it seems
+  // less error prone to (double) delete before destroying (noop if 
already deleted).
+  if (parent) {
+static_cast(parent)->connection_state.delete_stream(this);
+Warning("Http2Stream was about to be deallocated without removing it 
from the active stream list");
--- End diff --

We are sure this is not going to produce a lot of warnings, right ? :)


Issue Time Tracking
---

Worklog Id: (was: 30817)
Time Spent: 5h 40m  (was: 5.5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30816=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30816
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 17:27
Start Date: 18/Oct/16 17:27
Worklog Time Spent: 10m 
  Work Description: Github user bryancall commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
 - Looks good


Issue Time Tracking
---

Worklog Id: (was: 30816)
Time Spent: 5.5h  (was: 5h 20m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840
> --- count=2 ---
> id=19
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30815
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 17:24
Start Date: 18/Oct/16 17:24
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1117
  
Linux build *failed*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/932/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30815)
Time Spent: 5h 20m  (was: 5h 10m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30814
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 18/Oct/16 17:21
Start Date: 18/Oct/16 17:21
Worklog Time Spent: 10m 
  Work Description: GitHub user gtenev opened a pull request:

https://github.com/apache/trafficserver/pull/1117

TS-4916 Add safety net to avoid H2-infinite-loop deadlock.

Current Http2ConnectionState implementation uses a memory pool for
instantiating streams and DLL<> stream_list for storing active streams.
Destroying a stream before deleting it from stream_list and then creating
a new one + reusing the same chunk from the memory pool right away always
leads to destroying the DLL structure (deadlocks, inconsistencies).
Added a safety net since the consequences are disastrous.
Until the design/implementation changes it seems less error prone
to (double) delete before destroying (noop if already deleted).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gtenev/trafficserver TS-4916-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/trafficserver/pull/1117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1117


commit ed72728800fb3a5387172644b1cc586b74c4f0a0
Author: Gancho Tenev 
Date:   2016-10-17T22:10:12Z

TS-4916 Add safety net to avoid H2-infinite-loop deadlock.

Current Http2ConnectionState implementation uses a memory pool for
instantiating streams and DLL<> stream_list for storing active streams.
Destroying a stream before deleting it from stream_list and then creating
a new one + reusing the same chunk from the memory pool right away always
leads to destroying the DLL structure (deadlocks, inconsistencies).
Added a safety net since the consequences are disastrous.
Until the design/implementation changes it seems less error prone
to (double) delete before destroying (noop if already deleted).




Issue Time Tracking
---

Worklog Id: (was: 30814)
Time Spent: 5h 10m  (was: 5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30704
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 14/Oct/16 23:15
Start Date: 14/Oct/16 23:15
Worklog Time Spent: 10m 
  Work Description: Github user gtenev commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
I was actually thinking to propose just adding a “catch-all-delete-stream” 
call in `destroy()` before `THREAD_FREE()` in 7.0.

It seems to me that the point of this Jira TS-4813 / this PR got somehow 
lost so I would like to reiterate.

Missing a stream deletion from `DLL<>` before calling `THREAD_FREE()` COULD 
lead to breaking the `DLL<>` and then if that happens `DLL<>` traversals ALWAYS 
ends up in infinite loop. 

Host became dysfunctional pretty often because of this (time to live 1-3 
days per host) and it took long time to debug/prove.

This is my (practical?) attempt to make sure we don’t get stuck in this 
debugging again regardless of past/current/future “delete stream” bugs (as long 
us we use `THREAD_ALLOC_INIT/THREAD_FREE` + `DLL<>` of course). 

I wanted to propose a "catch-all-delete-stream" safety net in both 6.2.1 
and 7.0.0 because I think it would be hard to guarantee that the next commit 
would not introduce this weakness again.

This is my first read of the H2 code so I can see how the proposed fix can 
be sub-optimal and will gladly change it, I also hope I provided enough 
reasoning for best results :).

I am pretty confident in my debugging data/conclusions but just by reading 
the code or by sorting through Jiras that “may help” I can only guess if the 
new 7.0 version or back-ports to 6.2.1 definitely fix the issue until I 
test/verify.

We could close this PR:
- If we are confident all “fixes in this area” from 7.0 can be back-ported 
to 6.2.1 and that the issue will be fixed 
- AND If adding the proposed safety net does not make any/enough sense

Cheers!




Issue Time Tracking
---

Worklog Id: (was: 30704)
Time Spent: 5h  (was: 4h 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30700
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 14/Oct/16 22:28
Start Date: 14/Oct/16 22:28
Worklog Time Spent: 10m 
  Work Description: Github user gtenev commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83512075
  
--- Diff: proxy/http2/Http2Stream.cc ---
@@ -267,10 +267,12 @@ Http2Stream::do_io_close(int /* flags */)
   // Make sure any trailing end of stream frames are sent
   // Ourselve will be removed at send_data_frames or closing 
connection phase
   static_cast(parent)->connection_state.send_data_frames(this);
+
+  // Make sure the stream is deleted at this point since next step is 
self destroy.
+  this->delete_stream();
--- End diff --

This is what actually made sure we delete the stream from the `DLL<>` 
before  triggering `destroy()` (before leaving `do_io_close()`).

In the version 6.2.1 we have `send_data_frames()` delete the stream from 
`DLL<>` on `HTTP2_STREAM_STATE_CLOSED`.

Then in a later version we added `HTTP2_STREAM_STATE_HALF_CLOSED_LOCAL`. 

What caused the destroying of `DLL<>` in our case was 
`HTTP2_STREAM_STATE_HALF_CLOSED_REMOTE` (which does not cause deletion in any 
version).

It seemed to me we have been always vulnerable to this problem despite the 
fixes. That is why I thought I would add this “catch-all-delete-stream” line 
here for all the current and future missing states. After this point we are 
going to `destroy()` the stream  regardless of its state anyway.



Issue Time Tracking
---

Worklog Id: (was: 30700)
Time Spent: 4h 50m  (was: 4h 40m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30698
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 14/Oct/16 22:19
Start Date: 14/Oct/16 22:19
Worklog Time Spent: 10m 
  Work Description: Github user gtenev commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83511226
  
--- Diff: proxy/http2/Http2ConnectionState.cc ---
@@ -1097,6 +1141,7 @@ void
 Http2ConnectionState::send_data_frames(Http2Stream *stream)
 {
   if (stream->get_state() == HTTP2_STREAM_STATE_CLOSED) {
+this->delete_stream(stream);
 return;
--- End diff --

No, I may have overdone it here. 

I thought I would add this deletion because few lines below there is 
deleting if state == `HTTP2_STREAM_STATE_CLOSED`. `send_data_frames()` is 
called not only from `do_io_close()`, and calling `delete_stream()` on already 
deleted stream is OK (with this change).



Issue Time Tracking
---

Worklog Id: (was: 30698)
Time Spent: 4h 40m  (was: 4.5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30697=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30697
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 14/Oct/16 22:15
Start Date: 14/Oct/16 22:15
Worklog Time Spent: 10m 
  Work Description: Github user gtenev commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83510837
  
--- Diff: proxy/http2/Http2ConnectionState.cc ---
@@ -936,30 +940,70 @@ Http2ConnectionState::cleanup_streams()
 void
 Http2ConnectionState::delete_stream(Http2Stream *stream)
 {
+  // The following check allows the method to be called safely on already 
deleted streams.
+  if (deleted_from_active_streams(stream)) {
+return;
+  }
+
+  SCOPED_MUTEX_LOCK(lock, this->mutex, this_ethread());
+
--- End diff --

If we are sure `DLL<>` is always protected by a lock then I must have 
really misunderstood this previous [comment on 
TS-4916](https://issues.apache.org/jira/browse/TS-4916?focusedCommentId=15552505=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15552505])
 where we suspected `DLL<>` to be “manipulated by simultaneous threads”.

That would mean that at least in one thread was not holding the right lock. 
In that case would not that mean that “rearranging some of the stream_count 
book keeping” would rather hide the problem than to fix it?

Trying to help based on the comment decided to trace few paths in the 
source code that may not be holding ConnectionState lock (theoretically) and 
grabbed the lock on the common path closest to the structures that needed 
protection (which based on my understanding should not be a problem if the 
thread is already holding the lock).

Actually never noticed the race condition in my debugging so I am going to 
remove this line from the PR and I will consider limiting my future changes to 
the issue I am trying to fix.



Issue Time Tracking
---

Worklog Id: (was: 30697)
Time Spent: 4.5h  (was: 4h 20m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30694=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30694
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 14/Oct/16 21:55
Start Date: 14/Oct/16 21:55
Worklog Time Spent: 10m 
  Work Description: Github user gtenev commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83508421
  
--- Diff: proxy/http2/Http2ConnectionState.cc ---
@@ -936,30 +940,70 @@ Http2ConnectionState::cleanup_streams()
 void
 Http2ConnectionState::delete_stream(Http2Stream *stream)
 {
+  // The following check allows the method to be called safely on already 
deleted streams.
+  if (deleted_from_active_streams(stream)) {
+return;
+  }
+
--- End diff --

OK, needed that for calling `delete_stream()` on already deleted streams 
for my “catch-all-delete-stream” calls (6.2.1). 

Also grouped some of the `DLL<>` + `client_streams_count` updates in 
separate functions.

Looking at how the master changed since I started working on this I may 
need to revert that to be consistent with master.



Issue Time Tracking
---

Worklog Id: (was: 30694)
Time Spent: 4h 20m  (was: 4h 10m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30610
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 13/Oct/16 19:29
Start Date: 13/Oct/16 19:29
Worklog Time Spent: 10m 
  Work Description: Github user shinrich commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Not clear we want to move this to 7.0.  The main changes have already been 
applied (check to see if stream is still in list before deleting and 
appropriately signal that the stream should be removed from the 
Session/ConnState stream list).  May want to look at the backport of TS-4507 to 
6.2.


Issue Time Tracking
---

Worklog Id: (was: 30610)
Time Spent: 4h 10m  (was: 4h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30606
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 13/Oct/16 19:16
Start Date: 13/Oct/16 19:16
Worklog Time Spent: 10m 
  Work Description: Github user shinrich commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83290881
  
--- Diff: proxy/http2/Http2ConnectionState.cc ---
@@ -936,30 +940,70 @@ Http2ConnectionState::cleanup_streams()
 void
 Http2ConnectionState::delete_stream(Http2Stream *stream)
 {
+  // The following check allows the method to be called safely on already 
deleted streams.
+  if (deleted_from_active_streams(stream)) {
+return;
+  }
+
+  SCOPED_MUTEX_LOCK(lock, this->mutex, this_ethread());
+
--- End diff --

Why do you need to grab a lock here?  Given the mutex sharing between vc 
and session and ConnectionState, we should already be holding this mutex.


Issue Time Tracking
---

Worklog Id: (was: 30606)
Time Spent: 3h 40m  (was: 3.5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30608
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 13/Oct/16 19:18
Start Date: 13/Oct/16 19:18
Worklog Time Spent: 10m 
  Work Description: Github user shinrich commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83291283
  
--- Diff: proxy/http2/Http2Stream.cc ---
@@ -267,10 +267,12 @@ Http2Stream::do_io_close(int /* flags */)
   // Make sure any trailing end of stream frames are sent
   // Ourselve will be removed at send_data_frames or closing 
connection phase
   static_cast(parent)->connection_state.send_data_frames(this);
+
+  // Make sure the stream is deleted at this point since next step is 
self destroy.
+  this->delete_stream();
--- End diff --

It isn't clear to me that this delete_stream is necessary either.


Issue Time Tracking
---

Worklog Id: (was: 30608)
Time Spent: 4h  (was: 3h 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30607
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 13/Oct/16 19:17
Start Date: 13/Oct/16 19:17
Worklog Time Spent: 10m 
  Work Description: Github user shinrich commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83291135
  
--- Diff: proxy/http2/Http2ConnectionState.cc ---
@@ -1097,6 +1141,7 @@ void
 Http2ConnectionState::send_data_frames(Http2Stream *stream)
 {
   if (stream->get_state() == HTTP2_STREAM_STATE_CLOSED) {
+this->delete_stream(stream);
 return;
--- End diff --

Were we leaking a stream without a delete call? 


Issue Time Tracking
---

Worklog Id: (was: 30607)
Time Spent: 3h 50m  (was: 3h 40m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30601
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 13/Oct/16 18:20
Start Date: 13/Oct/16 18:20
Worklog Time Spent: 10m 
  Work Description: Github user gtenev commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83279595
  
--- Diff: iocore/aio/.diags.log.meta ---
@@ -0,0 +1 @@
+creation_time = 1476307057
--- End diff --

removed, committed by mistake


Issue Time Tracking
---

Worklog Id: (was: 30601)
Time Spent: 3.5h  (was: 3h 20m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30592
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 13/Oct/16 17:18
Start Date: 13/Oct/16 17:18
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Linux build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/892/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30592)
Time Spent: 3h 20m  (was: 3h 10m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30591
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 13/Oct/16 17:18
Start Date: 13/Oct/16 17:18
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/1000/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30591)
Time Spent: 3h 10m  (was: 3h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30567=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30567
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 13/Oct/16 07:14
Start Date: 13/Oct/16 07:14
Worklog Time Spent: 10m 
  Work Description: Github user masaori335 commented on a diff in the pull 
request:

https://github.com/apache/trafficserver/pull/1100#discussion_r83155537
  
--- Diff: iocore/aio/.diags.log.meta ---
@@ -0,0 +1 @@
+creation_time = 1476307057
--- End diff --

Do we need this file?


Issue Time Tracking
---

Worklog Id: (was: 30567)
Time Spent: 3h  (was: 2h 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30557
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 23:38
Start Date: 12/Oct/16 23:38
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Linux build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/882/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30557)
Time Spent: 2h 50m  (was: 2h 40m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30556
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 23:35
Start Date: 12/Oct/16 23:35
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/990/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30556)
Time Spent: 2h 40m  (was: 2.5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30546=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30546
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 22:23
Start Date: 12/Oct/16 22:23
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/987/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30546)
Time Spent: 2.5h  (was: 2h 20m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30545
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 22:13
Start Date: 12/Oct/16 22:13
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Linux build *failed*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/879/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30545)
Time Spent: 2h 20m  (was: 2h 10m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30542
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 22:06
Start Date: 12/Oct/16 22:06
Worklog Time Spent: 10m 
  Work Description: Github user zwoop commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Try again [approve ci].


Issue Time Tracking
---

Worklog Id: (was: 30542)
Time Spent: 2h 10m  (was: 2h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840
> --- count=2 ---
> id=19
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30540
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 22:00
Start Date: 12/Oct/16 22:00
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/986/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30540)
Time Spent: 2h  (was: 1h 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30539
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 22:00
Start Date: 12/Oct/16 22:00
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Linux build *failed*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/878/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30539)
Time Spent: 2h  (was: 1h 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30536
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 21:48
Start Date: 12/Oct/16 21:48
Worklog Time Spent: 10m 
  Work Description: Github user zwoop commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Try again [approve ci].


Issue Time Tracking
---

Worklog Id: (was: 30536)
Time Spent: 1h 50m  (was: 1h 40m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840
> --- count=2 ---
> id=19
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30532=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30532
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 21:16
Start Date: 12/Oct/16 21:16
Worklog Time Spent: 10m 
  Work Description: Github user zwoop commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Just a heads up, but this PR is against 6.2.x branch, the code has 
diverged, so will need one PR for 6.2.x and one for master. @gtenev Please make 
a PR for master at your earliest convenience as well, but sounds like they will 
be so different that we need two reviews.


Issue Time Tracking
---

Worklog Id: (was: 30532)
Time Spent: 1h 40m  (was: 1.5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30531
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 21:10
Start Date: 12/Oct/16 21:10
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/984/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30531)
Time Spent: 1.5h  (was: 1h 20m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30530
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 21:08
Start Date: 12/Oct/16 21:08
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1100
  
Linux build *failed*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/876/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30530)
Time Spent: 1h 20m  (was: 1h 10m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30529
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 20:59
Start Date: 12/Oct/16 20:59
Worklog Time Spent: 10m 
  Work Description: GitHub user gtenev opened a pull request:

https://github.com/apache/trafficserver/pull/1100

TS-4916: Fix for an H2-infinite-loop deadlock

This is a fix to prevent destroying of the DLL<> structure and the 
following iteration over Http2ConnectionState::stream_list to fall into an 
infinite loop while holding a lock, which leads to cache updates to start 
failing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gtenev/trafficserver TS-4916

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/trafficserver/pull/1100.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1100


commit e84978a18f0d856adfea744a5ab11668e1fc
Author: Gancho Tenev 
Date:   2016-10-11T23:05:14Z

TS-4916 Fix for a H2-infinite-loop deadlock.




Issue Time Tracking
---

Worklog Id: (was: 30529)
Time Spent: 1h 10m  (was: 1h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30528=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30528
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 20:56
Start Date: 12/Oct/16 20:56
Worklog Time Spent: 10m 
  Work Description: Github user gtenev closed the pull request at:

https://github.com/apache/trafficserver/pull/1099


Issue Time Tracking
---

Worklog Id: (was: 30528)
Time Spent: 1h  (was: 50m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840
> --- count=2 ---
> id=19
> this=0x2ae5b6bbec00
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30527
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 20:56
Start Date: 12/Oct/16 20:56
Worklog Time Spent: 10m 
  Work Description: Github user gtenev commented on the issue:

https://github.com/apache/trafficserver/pull/1099
  
@zwoop I did, my fork looks OK. it seems to me that later in the github GUI 
I may have submitted the PR against base:master (which was the default), should 
have been against 6.2.x. Closing.


Issue Time Tracking
---

Worklog Id: (was: 30527)
Time Spent: 50m  (was: 40m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30525=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30525
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 20:25
Start Date: 12/Oct/16 20:25
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1099
  
Linux build *failed*! See 
https://ci.trafficserver.apache.org/job/Github-Linux/875/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30525)
Time Spent: 0.5h  (was: 20m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> 

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30526=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30526
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 20:29
Start Date: 12/Oct/16 20:29
Worklog Time Spent: 10m 
  Work Description: Github user atsci commented on the issue:

https://github.com/apache/trafficserver/pull/1099
  
FreeBSD build *successful*! See 
https://ci.trafficserver.apache.org/job/Github-FreeBSD/983/ for details.
 



Issue Time Tracking
---

Worklog Id: (was: 30526)
Time Spent: 40m  (was: 0.5h)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30524
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 20:16
Start Date: 12/Oct/16 20:16
Worklog Time Spent: 10m 
  Work Description: Github user zwoop commented on the issue:

https://github.com/apache/trafficserver/pull/1099
  
@gtenev This looks very strange, did you pull/rebase your tree first?


Issue Time Tracking
---

Worklog Id: (was: 30524)
Time Spent: 20m  (was: 10m)

> Http2ConnectionState::restart_streams infinite loop causes deadlock 
> 
>
> Key: TS-4916
> URL: https://issues.apache.org/jira/browse/TS-4916
> Project: Traffic Server
>  Issue Type: Bug
>  Components: Core, HTTP/2
>Reporter: Gancho Tenev
>Assignee: Gancho Tenev
>Priority: Blocker
> Fix For: 7.1.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Http2ConnectionState::restart_streams falls into an infinite loop while 
> holding a lock, which leads to cache updates to start failing.
> The infinite loop is caused by traversing a list whose last element “next” 
> points to the element itself and the traversal never finishes.
> {code}
> Thread 51 (Thread 0x2aaab3d04700 (LWP 34270)):
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> #1  rcv_window_update_frame (cstate=..., frame=...) at 
> Http2ConnectionState.cc:627
> #2  0x2acf9738 in Http2ConnectionState::main_event_handler 
> (this=0x2ae6ba5284c8, event=, edata=) at 
> Http2ConnectionState.cc:823
> #3  0x2acef1c3 in Continuation::handleEvent (data=0x2aaab3d039a0, 
> event=2253, this=0x2ae6ba5284c8) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #4  send_connection_event (cont=cont@entry=0x2ae6ba5284c8, 
> event=event@entry=2253, edata=edata@entry=0x2aaab3d039a0) at 
> Http2ClientSession.cc:58
> #5  0x2acef462 in Http2ClientSession::state_complete_frame_read 
> (this=0x2ae6ba528290, event=, edata=0x2aab7b237f18) at 
> Http2ClientSession.cc:426
> #6  0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #7  Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #8  0x2acef5a3 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #9  Http2ClientSession::state_complete_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:431
> #10 0x2acf0982 in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=0x2ae6ba528290) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #11 Http2ClientSession::state_start_frame_read (this=0x2ae6ba528290, 
> event=, edata=0x2aab7b237f18) at Http2ClientSession.cc:399
> #12 0x2ae67e2b in Continuation::handleEvent (data=0x2aab7b237f18, 
> event=100, this=) at 
> ../../iocore/eventsystem/I_Continuation.h:153
> #13 read_signal_and_update (vc=0x2aab7b237e00, vc@entry=0x1, 
> event=event@entry=100) at UnixNetVConnection.cc:153
> #14 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2aab7b237e00, 
> event=event@entry=100) at UnixNetVConnection.cc:1036
> #15 0x2ae47653 in SSLNetVConnection::net_read_io 
> (this=0x2aab7b237e00, nh=0x2aaab2409cc0, lthread=0x2aaab2406000) at 
> SSLNetVConnection.cc:595
> #16 0x2ae5558c in NetHandler::mainNetEvent (this=0x2aaab2409cc0, 
> event=, e=) at UnixNet.cc:513
> #17 0x2ae8d2e6 in Continuation::handleEvent (data=0x2aaab0bfa700, 
> event=5, this=) at I_Continuation.h:153
> #18 EThread::process_event (calling_code=5, e=0x2aaab0bfa700, 
> this=0x2aaab2406000) at UnixEThread.cc:148
> #19 EThread::execute (this=0x2aaab2406000) at UnixEThread.cc:275
> #20 0x2ae8c0e6 in spawn_thread_internal (a=0x2aaab0b25bb0) at 
> Thread.cc:86
> #21 0x2d6b3aa1 in start_thread (arg=0x2aaab3d04700) at 
> pthread_create.c:301
> #22 0x2e8bc93d in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> {code}
> Here is the stream_list trace.
> {code}
> (gdb) thread 51
> [Switching to thread 51 (Thread 0x2aaab3d04700 (LWP 34270))]
> #0  0x2acf3fee in Http2ConnectionState::restart_streams 
> (this=0x2ae6ba5284c8) at Http2ConnectionState.cc:913
> (gdb) trace_list stream_list
> --- count=0 ---
> id=29
> this=0x2ae673f0c840
> next=0x2aaac05d8900
> prev=(nil)
> --- count=1 ---
> id=27
> this=0x2aaac05d8900
> next=0x2ae5b6bbec00
> prev=0x2ae673f0c840

[jira] [Work logged] (TS-4916) Http2ConnectionState::restart_streams infinite loop causes deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TS-4916?focusedWorklogId=30522=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-30522
 ]

ASF GitHub Bot logged work on TS-4916:
--

Author: ASF GitHub Bot
Created on: 12/Oct/16 20:13
Start Date: 12/Oct/16 20:13
Worklog Time Spent: 10m 
  Work Description: GitHub user gtenev opened a pull request:

https://github.com/apache/trafficserver/pull/1099

TS-4916: Fix for an H2-infinite-loop deadlock

This is a fix to prevent destroying of the ``DLL<>`` structure and the 
following iteration over ``Http2ConnectionState::stream_list`` to fall into an 
infinite loop while holding a lock, which leads to cache updates to start 
failing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gtenev/trafficserver TS-4916

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/trafficserver/pull/1099.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1099


commit d9b5237255ab86d9013a29562db6b839122cec23
Author: Phil Sorber 
Date:   2016-04-18T04:58:51Z

Mark proxy.config.ssl.SSLv2 and proxy.config.ssl.SSLv3 as deprecated in the 
docs.

commit bad058eb152772f1df0fe4c133f4ac62fa0eaa12
Author: Leif Hedstrom 
Date:   2016-04-04T21:06:47Z

TS-4318 Fix a regression in regex rules

The refactoring done earlier broke the config loading of rules using
the regular expressions. This restore that functionality, but cleaner.

(cherry picked from commit 431a8f838e75338cb685b95c213a6140f5cbdcc7)

commit 8261836994bc74fd3f1a078c976082e91b4bc976
Author: Meera Mosale Nataraja 
Date:   2016-04-18T19:51:03Z

TS-4147 Allow gzip plugin to be a remap plugin

(cherry picked from commit 3d7cdbfa1dc52e81c2698dfbac526dec1d336246)

commit 12084df688483d939654dd53d3c82e5f3cc9
Author: Masaori Koshiba 
Date:   2016-04-18T06:16:56Z

TS-4361: Remove TS_FETCH_EVENT handlers from Http2ClientSession. This 
closes #581.

(cherry picked from commit 3872a385cac3d43de25bdaf38c56bbcc2cb6f65b)

commit 377ffdc65978dd10344dd4a3bcef0677f8263a4e
Author: shinrich 
Date:   2016-04-19T14:40:20Z

TS-4364: Address remnants from HTTP/2 refactoring.  This closes #584.

(cherry picked from commit 2a1c3996d1641bd246faf39a171b330ac33da34c)

commit 556a0f9717eb10cdd20d8659bc605bdb10c27ad0
Author: shinrich 
Date:   2016-04-18T15:15:16Z

TS-3429: Fix reference counting for TSContScheduleEvery.  This closes #576.

(cherry picked from commit 3286600d90be4b6291b35867720adbd864f580ea)

commit 7bf73303bf325b83492c43b58d9241e20a71c475
Author: Masaori Koshiba 
Date:   2016-04-25T02:12:17Z

TS-4355: Change assert condition for TS-3612

Originally, the condition of assert in HttpTunnel::main_handler(int event, 
void *data)
was same to the condition in HttpTunnel::get_consumer(VIO *vio) before 
TS-3612.
This commit makes the condition in HttpTunnel::main_handler(int event, void 
*data)
same as condition in HttpTunnel::get_consumer(VIO *vio).

(cherry picked from commit 37f7c05de574458d2eff781d9f047673cda97a4e)

commit c754bc35de1ce5d5984990ab07f5ef8588743615
Author: shinrich 
Date:   2016-04-26T20:33:10Z

TS-4378:  Remove periodic warning.

(cherry picked from commit 13a20c199803aae734b634da425340c72cf10b26)

commit af91bf80d8a6ca1a85056c1ff960f8848b7dfac3
Author: Thomas Jackson 
Date:   2016-05-02T15:34:05Z

TS-4403: Fix stale-while-revalidate on DNS lookup failures (#609)

HostDB's "stale-while-revalidate" feature allows hostdb to return stale 
records while doing the DNS lookup in the background. This works properly in 
the case where the resolver goes away, but in the case that an error was 
returned from the resolver the record in cache was thrown away. This means that 
a transient error out in the DNS infrastructure would cause ATS to drop its 
stale record it would have contently served-- this patch simply makes hostdb 
honor its stale-while-revalidate contract (if configured). So, in the event 
that the DNS result comes back as `failed` hostdb will keep the old one if it 
is okay with being served stale.

This closes #609
(cherry picked from commit 131875cbe7b0975c8c3c4f3d20aa55a5c7caba86)

commit b5317c9ee8812f6be65c0b7f215393e62fb6fe2e
Author: Felix Bünemann 
Date:   2016-04-30T06:09:24Z

TS-4397: Fix build on i386 caused by type mismatch

It seems lua_Integer is 32-Bit on i386 while it's 54-Bit on x86-64
causing the existing code to fail with: