[jira] [Commented] (PROTON-1171) [proton-j] transport SSL wrapper does not flush all decoded bytes to the underlying input
[ https://issues.apache.org/jira/browse/PROTON-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242230#comment-15242230 ] ASF GitHub Bot commented on PROTON-1171: Github user SreeramGarlapati commented on the pull request: https://github.com/apache/qpid-proton/pull/73#issuecomment-210225444 After a couple of hours of our Stress run - we are still experiencing some bytes stuck in this layer. Have sent you a mail on the proton forum. Thx! > [proton-j] transport SSL wrapper does not flush all decoded bytes to the > underlying input > - > > Key: PROTON-1171 > URL: https://issues.apache.org/jira/browse/PROTON-1171 > Project: Qpid Proton > Issue Type: Bug > Components: proton-j >Affects Versions: 0.12.0, 0.12.1 > Environment: Any >Reporter: Sreeram Garlapati >Assignee: Robbie Gemmell >Priority: Critical > Fix For: 0.13.0, 0.12.2 > > > Actual Issue/scenario hit by Microsoft Azure EventHubs: > We have a pattern where customers sends messages in a burst to our Queue and > stop sending and then wait for all of them to be received. > Because of this issue in Proton-j Amqp implementation - we can see many bytes > were stuck in the SSL Decode Buffer and were only picked up when new > transport frames arrive. > Given that this is a 1 line fix - it would be great if you folks can publish > an incremental minor update to 0.12.X. > Here are my findings after debugging through this issue: > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request: > https://github.com/apache/qpid-proton/pull/73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-1171) [proton-j] transport SSL wrapper does not flush all decoded bytes to the underlying input
[ https://issues.apache.org/jira/browse/PROTON-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240917#comment-15240917 ] ASF subversion and git services commented on PROTON-1171: - Commit 91aff1316bbbc7941a6f5fe8d3e0366744896186 in qpid-proton's branch refs/heads/0.12.x from Robert Gemmell [ https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=91aff13 ] PROTON-1171: update transport ssl wrapper to ensure attempt to pass all decoded bytes to the underlying input or throw, and avoid processing the underlying layer when no bytes have been passed on This closes #73 (cherry picked from commit 3ea4287d483f3ce275be75bf50ac1728527d697a) > [proton-j] transport SSL wrapper does not flush all decoded bytes to the > underlying input > - > > Key: PROTON-1171 > URL: https://issues.apache.org/jira/browse/PROTON-1171 > Project: Qpid Proton > Issue Type: Bug > Components: proton-j >Affects Versions: 0.12.0, 0.12.1 > Environment: Any >Reporter: Sreeram Garlapati >Assignee: Robbie Gemmell >Priority: Critical > Fix For: 0.13.0, 0.12.2 > > > Actual Issue/scenario hit by Microsoft Azure EventHubs: > We have a pattern where customers sends messages in a burst to our Queue and > stop sending and then wait for all of them to be received. > Because of this issue in Proton-j Amqp implementation - we can see many bytes > were stuck in the SSL Decode Buffer and were only picked up when new > transport frames arrive. > Given that this is a 1 line fix - it would be great if you folks can publish > an incremental minor update to 0.12.X. > Here are my findings after debugging through this issue: > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request: > https://github.com/apache/qpid-proton/pull/73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-1171) [proton-j] transport SSL wrapper does not flush all decoded bytes to the underlying input
[ https://issues.apache.org/jira/browse/PROTON-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240918#comment-15240918 ] ASF subversion and git services commented on PROTON-1171: - Commit a7d3c52bc24d05bc21236a2304151e4d7050 in qpid-proton's branch refs/heads/0.12.x from Robert Gemmell [ https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=a7d3c52 ] PROTON-1171: correct comment, remove leftover System.out.println (cherry picked from commit c22a25cd890d0e28a52a36a59bf02b7f900037e2) > [proton-j] transport SSL wrapper does not flush all decoded bytes to the > underlying input > - > > Key: PROTON-1171 > URL: https://issues.apache.org/jira/browse/PROTON-1171 > Project: Qpid Proton > Issue Type: Bug > Components: proton-j >Affects Versions: 0.12.0, 0.12.1 > Environment: Any >Reporter: Sreeram Garlapati >Assignee: Robbie Gemmell >Priority: Critical > Fix For: 0.13.0, 0.12.2 > > > Actual Issue/scenario hit by Microsoft Azure EventHubs: > We have a pattern where customers sends messages in a burst to our Queue and > stop sending and then wait for all of them to be received. > Because of this issue in Proton-j Amqp implementation - we can see many bytes > were stuck in the SSL Decode Buffer and were only picked up when new > transport frames arrive. > Given that this is a 1 line fix - it would be great if you folks can publish > an incremental minor update to 0.12.X. > Here are my findings after debugging through this issue: > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request: > https://github.com/apache/qpid-proton/pull/73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-1171) [proton-j] transport SSL wrapper does not flush all decoded bytes to the underlying input
[ https://issues.apache.org/jira/browse/PROTON-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240897#comment-15240897 ] ASF subversion and git services commented on PROTON-1171: - Commit c22a25cd890d0e28a52a36a59bf02b7f900037e2 in qpid-proton's branch refs/heads/master from Robert Gemmell [ https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=c22a25c ] PROTON-1171: correct comment, remove leftover System.out.println > [proton-j] transport SSL wrapper does not flush all decoded bytes to the > underlying input > - > > Key: PROTON-1171 > URL: https://issues.apache.org/jira/browse/PROTON-1171 > Project: Qpid Proton > Issue Type: Bug > Components: proton-j >Affects Versions: 0.12.0, 0.12.1 > Environment: Any >Reporter: Sreeram Garlapati >Assignee: Robbie Gemmell >Priority: Critical > Fix For: 0.13.0, 0.12.2 > > > Actual Issue/scenario hit by Microsoft Azure EventHubs: > We have a pattern where customers sends messages in a burst to our Queue and > stop sending and then wait for all of them to be received. > Because of this issue in Proton-j Amqp implementation - we can see many bytes > were stuck in the SSL Decode Buffer and were only picked up when new > transport frames arrive. > Given that this is a 1 line fix - it would be great if you folks can publish > an incremental minor update to 0.12.X. > Here are my findings after debugging through this issue: > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request: > https://github.com/apache/qpid-proton/pull/73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-1171) [proton-j] transport SSL wrapper does not flush all decoded bytes to the underlying input
[ https://issues.apache.org/jira/browse/PROTON-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239886#comment-15239886 ] ASF GitHub Bot commented on PROTON-1171: Github user SreeramGarlapati commented on the pull request: https://github.com/apache/qpid-proton/pull/73#issuecomment-209605630 This is huge. Thanks @gemmellr ! We validated this in our stress run - you are correct - with my fix - we were still hitting another issue - which I was still looking at. After I packed proton with your commit - we are completely unblocked from receive stuck issue. Looking forward for 0.12.2. > [proton-j] transport SSL wrapper does not flush all decoded bytes to the > underlying input > - > > Key: PROTON-1171 > URL: https://issues.apache.org/jira/browse/PROTON-1171 > Project: Qpid Proton > Issue Type: Bug > Components: proton-j >Affects Versions: 0.12.0, 0.12.1 > Environment: Any >Reporter: Sreeram Garlapati >Assignee: Robbie Gemmell >Priority: Critical > Fix For: 0.13.0, 0.12.2 > > > Actual Issue/scenario hit by Microsoft Azure EventHubs: > We have a pattern where customers sends messages in a burst to our Queue and > stop sending and then wait for all of them to be received. > Because of this issue in Proton-j Amqp implementation - we can see many bytes > were stuck in the SSL Decode Buffer and were only picked up when new > transport frames arrive. > Given that this is a 1 line fix - it would be great if you folks can publish > an incremental minor update to 0.12.X. > Here are my findings after debugging through this issue: > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request: > https://github.com/apache/qpid-proton/pull/73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-1171) [proton-j] transport SSL wrapper does not flush all decoded bytes to the underlying input
[ https://issues.apache.org/jira/browse/PROTON-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239286#comment-15239286 ] ASF GitHub Bot commented on PROTON-1171: Github user gemmellr commented on the pull request: https://github.com/apache/qpid-proton/pull/73#issuecomment-209455781 I have made a change that should resolve this. Its a little different than your proposal to both resolve an issue with it (unlikely but possible infinite loop), and also change some other existing behaviours I found while digging into it. If you could give it a try with your bits and report back that would be great. > [proton-j] transport SSL wrapper does not flush all decoded bytes to the > underlying input > - > > Key: PROTON-1171 > URL: https://issues.apache.org/jira/browse/PROTON-1171 > Project: Qpid Proton > Issue Type: Bug > Components: proton-j >Affects Versions: 0.12.0, 0.12.1 > Environment: Any >Reporter: Sreeram Garlapati >Assignee: Robbie Gemmell >Priority: Critical > Fix For: 0.13.0, 0.12.2 > > > Actual Issue/scenario hit by Microsoft Azure EventHubs: > We have a pattern where customers sends messages in a burst to our Queue and > stop sending and then wait for all of them to be received. > Because of this issue in Proton-j Amqp implementation - we can see many bytes > were stuck in the SSL Decode Buffer and were only picked up when new > transport frames arrive. > Given that this is a 1 line fix - it would be great if you folks can publish > an incremental minor update to 0.12.X. > Here are my findings after debugging through this issue: > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request: > https://github.com/apache/qpid-proton/pull/73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-1171) [proton-j] transport SSL wrapper does not flush all decoded bytes to the underlying input
[ https://issues.apache.org/jira/browse/PROTON-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239273#comment-15239273 ] ASF GitHub Bot commented on PROTON-1171: Github user asfgit closed the pull request at: https://github.com/apache/qpid-proton/pull/73 > [proton-j] transport SSL wrapper does not flush all decoded bytes to the > underlying input > - > > Key: PROTON-1171 > URL: https://issues.apache.org/jira/browse/PROTON-1171 > Project: Qpid Proton > Issue Type: Bug > Components: proton-j >Affects Versions: 0.12.0, 0.12.1 > Environment: Any >Reporter: Sreeram Garlapati >Assignee: Robbie Gemmell >Priority: Critical > Fix For: 0.13.0, 0.12.2 > > > Actual Issue/scenario hit by Microsoft Azure EventHubs: > We have a pattern where customers sends messages in a burst to our Queue and > stop sending and then wait for all of them to be received. > Because of this issue in Proton-j Amqp implementation - we can see many bytes > were stuck in the SSL Decode Buffer and were only picked up when new > transport frames arrive. > Given that this is a 1 line fix - it would be great if you folks can publish > an incremental minor update to 0.12.X. > Here are my findings after debugging through this issue: > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request: > https://github.com/apache/qpid-proton/pull/73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-1171) [proton-j] transport SSL wrapper does not flush all decoded bytes to the underlying input
[ https://issues.apache.org/jira/browse/PROTON-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239272#comment-15239272 ] ASF subversion and git services commented on PROTON-1171: - Commit 3ea4287d483f3ce275be75bf50ac1728527d697a in qpid-proton's branch refs/heads/master from Robert Gemmell [ https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=3ea4287 ] PROTON-1171: update transport ssl wrapper to ensure attempt to pass all decoded bytes to the underlying input or throw, and avoid processing the underlying layer when no bytes have been passed on This closes #73 > [proton-j] transport SSL wrapper does not flush all decoded bytes to the > underlying input > - > > Key: PROTON-1171 > URL: https://issues.apache.org/jira/browse/PROTON-1171 > Project: Qpid Proton > Issue Type: Bug > Components: proton-j >Affects Versions: 0.12.0, 0.12.1 > Environment: Any >Reporter: Sreeram Garlapati >Assignee: Robbie Gemmell >Priority: Critical > Fix For: 0.13.0, 0.12.2 > > > Actual Issue/scenario hit by Microsoft Azure EventHubs: > We have a pattern where customers sends messages in a burst to our Queue and > stop sending and then wait for all of them to be received. > Because of this issue in Proton-j Amqp implementation - we can see many bytes > were stuck in the SSL Decode Buffer and were only picked up when new > transport frames arrive. > Given that this is a 1 line fix - it would be great if you folks can publish > an incremental minor update to 0.12.X. > Here are my findings after debugging through this issue: > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request: > https://github.com/apache/qpid-proton/pull/73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
