RE: Proton-j Reactor - Receiver
Hello Robbie – That’s correct, please go ahead with the release 0.12.2. As per your suggestion, I will follow up on this issue with a Simple reproducer. Thx! Sree From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> Sent: Friday, April 15, 2016 7:09 AM To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; hm...@microsoft.com<mailto:hm...@microsoft.com> Subject: Re: Proton-j Reactor - Receiver Hi Sree, Your attachment didnt make it, the mailing list strips almost all attachments, particularly of any non trivial size. Best to attach stuff to JIRAs or provide links to them elsewhere. I had another look at it and still don't see a way for it to do what it did before, I'd guess it may be something slightly different. I think I'll need more analysis and/or a reproducer from you to make any further progress. 0.12.2 currently has the votes to pass. If it stays that way I probably lean towards proceeding with its release unless more concrete details becomes obvious, since it does seem to give a good improvement over the previous behaviour. Robbie On 15 April 2016 at 01:35, Garlapati Sreeram Kumar wrote: > Hello Robbie, > > > > After couple hours of Stress Run with the proton-j change – we still could > repro the Receive Stuck problem. Although – the below bug fix places us in a > much better state now. > > Attached a screenshot of Objects sizes on Heap – which corresponds to the > codepath that you fixed (SimpleSslTransportWrapper). > > Please see if this rings any bells – I am more than happy to share more > details (proton traces & dump – pls suggest if you will need more details). > > > > Thx! > > Sree > > > > From: Garlapati Sreeram Kumar > Sent: Monday, April 11, 2016 11:50 AM > To: Robbie Gemmell; proton@qpid.apache.org > Cc: SeongJoon Kwak (SJ); hm...@microsoft.com > Subject: RE: Proton-j Reactor - Receiver > > > > Awesome. > > To make it easy - added you as collaborator to my fork of Proton & here’s > the branch from which I submitted the PR: > https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck > > Thx! > Sree > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: Monday, April 11, 2016 9:52 AM > To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> > Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; > hm...@microsoft.com<mailto:hm...@microsoft.com> > Subject: Re: Proton-j Reactor - Receiver > > Ah, excellent. I had actually started on testing this myself a little > earlier, so I'll take a look and see whats what before continuing > tomorrow. On taking an initial better look at things I think the > change itself may need augmented to account for some other conditions > too, need to investigate further to be sure. > > Robbie > > On 11 April 2016 at 17:37, Garlapati Sreeram Kumar > wrote: >> Thanks a lot for the Response Robbie! >> Per your suggestion, added the CIT to the Pull Request (& yes, as you >> already said – this issue is being tracked via JIRA - PROTON-1171). >> >> Thanks a lot for the Wonderful Collaboration! >> Sree >> >> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> >> Sent: Thursday, April 7, 2016 3:52 AM >> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> >> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; >> hm...@microsoft.com<mailto:hm...@microsoft.com> >> Subject: Re: Proton-j Reactor - Receiver >> >> Hi Sree, >> >> Thanks for the analysis and PR, I'll try to take a proper look soon. >> It's not an area of the code I'm familiar with so I'll need to have a >> bit of a dig myself to see if the change seems ok. I'd note that any >> not-insignificant bug fix such as this should probably have a test >> with it (and a JIRA, though I see you have since created one of those) >> :) >> >> Robbie >> >> On 6 April 2016 at 01:23, Garlapati Sreeram Kumar >> wrote: >>> Hello Robbie, >>> >>> We are using proton-j client with SSL and many of our customers are >>> hitting this issue. >>> Here are my findings after debugging through this issue: >>> >>> - When incoming bytes arrive on the SocketChannel – proton-j >>> client gets signaled by nio & as a result it unwinds the transport stack – >>> as a result all the TransportInput implementations performs its task on the >>> Read Bytes and hands off to the Next Layer in the stack (transport to ssl, >>> ssl to frameparser etc). >>> >>> -
Re: Proton-j Reactor - Receiver
Hi Sree, Your attachment didnt make it, the mailing list strips almost all attachments, particularly of any non trivial size. Best to attach stuff to JIRAs or provide links to them elsewhere. I had another look at it and still don't see a way for it to do what it did before, I'd guess it may be something slightly different. I think I'll need more analysis and/or a reproducer from you to make any further progress. 0.12.2 currently has the votes to pass. If it stays that way I probably lean towards proceeding with its release unless more concrete details becomes obvious, since it does seem to give a good improvement over the previous behaviour. Robbie On 15 April 2016 at 01:35, Garlapati Sreeram Kumar wrote: > Hello Robbie, > > > > After couple hours of Stress Run with the proton-j change – we still could > repro the Receive Stuck problem. Although – the below bug fix places us in a > much better state now. > > Attached a screenshot of Objects sizes on Heap – which corresponds to the > codepath that you fixed (SimpleSslTransportWrapper). > > Please see if this rings any bells – I am more than happy to share more > details (proton traces & dump – pls suggest if you will need more details). > > > > Thx! > > Sree > > > > From: Garlapati Sreeram Kumar > Sent: Monday, April 11, 2016 11:50 AM > To: Robbie Gemmell; proton@qpid.apache.org > Cc: SeongJoon Kwak (SJ); hm...@microsoft.com > Subject: RE: Proton-j Reactor - Receiver > > > > Awesome. > > To make it easy - added you as collaborator to my fork of Proton & here’s > the branch from which I submitted the PR: > https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck > > Thx! > Sree > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: Monday, April 11, 2016 9:52 AM > To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> > Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; > hm...@microsoft.com<mailto:hm...@microsoft.com> > Subject: Re: Proton-j Reactor - Receiver > > Ah, excellent. I had actually started on testing this myself a little > earlier, so I'll take a look and see whats what before continuing > tomorrow. On taking an initial better look at things I think the > change itself may need augmented to account for some other conditions > too, need to investigate further to be sure. > > Robbie > > On 11 April 2016 at 17:37, Garlapati Sreeram Kumar > wrote: >> Thanks a lot for the Response Robbie! >> Per your suggestion, added the CIT to the Pull Request (& yes, as you >> already said – this issue is being tracked via JIRA - PROTON-1171). >> >> Thanks a lot for the Wonderful Collaboration! >> Sree >> >> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> >> Sent: Thursday, April 7, 2016 3:52 AM >> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> >> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; >> hm...@microsoft.com<mailto:hm...@microsoft.com> >> Subject: Re: Proton-j Reactor - Receiver >> >> Hi Sree, >> >> Thanks for the analysis and PR, I'll try to take a proper look soon. >> It's not an area of the code I'm familiar with so I'll need to have a >> bit of a dig myself to see if the change seems ok. I'd note that any >> not-insignificant bug fix such as this should probably have a test >> with it (and a JIRA, though I see you have since created one of those) >> :) >> >> Robbie >> >> On 6 April 2016 at 01:23, Garlapati Sreeram Kumar >> wrote: >>> Hello Robbie, >>> >>> We are using proton-j client with SSL and many of our customers are >>> hitting this issue. >>> Here are my findings after debugging through this issue: >>> >>> - When incoming bytes arrive on the SocketChannel – proton-j >>> client gets signaled by nio & as a result it unwinds the transport stack – >>> as a result all the TransportInput implementations performs its task on the >>> Read Bytes and hands off to the Next Layer in the stack (transport to ssl, >>> ssl to frameparser etc). >>> >>> - While unwinding that stack, >>> SimpleSSLTransportWrapper.unwrapInput reads(16k bytes) from _inputBuffer and >>> the result - decoded bytes are written to _decodedInputBuffer – as an >>> intermediate buffer. >>> >>> - It then flushes bytes from intermediate buffer to the next >>> layer & invokes an _underlyingInput.Process() – to signal it that it has >>> bytes in its input buffer. >>> >>> -
RE: Proton-j Reactor - Receiver
Hello Robbie, After couple hours of Stress Run with the proton-j change – we still could repro the Receive Stuck problem. Although – the below bug fix places us in a much better state now. Attached a screenshot of Objects sizes on Heap – which corresponds to the codepath that you fixed (SimpleSslTransportWrapper). Please see if this rings any bells – I am more than happy to share more details (proton traces & dump – pls suggest if you will need more details). Thx! Sree From: Garlapati Sreeram Kumar<mailto:sreer...@live.com> Sent: Monday, April 11, 2016 11:50 AM To: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>; proton@qpid.apache.org<mailto:proton@qpid.apache.org> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; hm...@microsoft.com<mailto:hm...@microsoft.com> Subject: RE: Proton-j Reactor - Receiver Awesome. To make it easy - added you as collaborator to my fork of Proton & here’s the branch from which I submitted the PR: https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck Thx! Sree From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> Sent: Monday, April 11, 2016 9:52 AM To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; hm...@microsoft.com<mailto:hm...@microsoft.com> Subject: Re: Proton-j Reactor - Receiver Ah, excellent. I had actually started on testing this myself a little earlier, so I'll take a look and see whats what before continuing tomorrow. On taking an initial better look at things I think the change itself may need augmented to account for some other conditions too, need to investigate further to be sure. Robbie On 11 April 2016 at 17:37, Garlapati Sreeram Kumar wrote: > Thanks a lot for the Response Robbie! > Per your suggestion, added the CIT to the Pull Request (& yes, as you already > said – this issue is being tracked via JIRA - PROTON-1171). > > Thanks a lot for the Wonderful Collaboration! > Sree > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: Thursday, April 7, 2016 3:52 AM > To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> > Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; > hm...@microsoft.com<mailto:hm...@microsoft.com> > Subject: Re: Proton-j Reactor - Receiver > > Hi Sree, > > Thanks for the analysis and PR, I'll try to take a proper look soon. > It's not an area of the code I'm familiar with so I'll need to have a > bit of a dig myself to see if the change seems ok. I'd note that any > not-insignificant bug fix such as this should probably have a test > with it (and a JIRA, though I see you have since created one of those) > :) > > Robbie > > On 6 April 2016 at 01:23, Garlapati Sreeram Kumar wrote: >> Hello Robbie, >> >> We are using proton-j client with SSL and many of our customers are hitting >> this issue. >> Here are my findings after debugging through this issue: >> >> - When incoming bytes arrive on the SocketChannel – proton-j client >> gets signaled by nio & as a result it unwinds the transport stack – as a >> result all the TransportInput implementations performs its task on the Read >> Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to >> frameparser etc). >> >> - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput >> reads(16k bytes) from _inputBuffer and the result - decoded bytes are >> written to _decodedInputBuffer – as an intermediate buffer. >> >> - It then flushes bytes from intermediate buffer to the next layer >> & invokes an _underlyingInput.Process() – to signal it that it has bytes in >> its input buffer. >> >> - If the underlyingInput (lets say FrameParser) buffer size is >> small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & >> Over time this accrues. >> >> The fix here is to flush decodedInputBuffer to the Next transport in the >> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer >> is empty. Here’s the pull request - >> https://github.com/apache/qpid-proton/pull/73 >> >> Pl. let me know if we need to do more to fix this issue comprehensively. >> >> Thx! >> Sree >> >> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> >> Sent: Thursday, March 31, 2016 9:19 AM >> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> >> Subject: Re: Proton-j Reactor - Receiver >> >> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar wrote: >>> Hello All! >>> >>> I am using Proton-J reactor API (Version
RE: Proton-j Reactor - Receiver
Awesome. To make it easy - added you as collaborator to my fork of Proton & here’s the branch from which I submitted the PR: https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck Thx! Sree From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> Sent: Monday, April 11, 2016 9:52 AM To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; hm...@microsoft.com<mailto:hm...@microsoft.com> Subject: Re: Proton-j Reactor - Receiver Ah, excellent. I had actually started on testing this myself a little earlier, so I'll take a look and see whats what before continuing tomorrow. On taking an initial better look at things I think the change itself may need augmented to account for some other conditions too, need to investigate further to be sure. Robbie On 11 April 2016 at 17:37, Garlapati Sreeram Kumar wrote: > Thanks a lot for the Response Robbie! > Per your suggestion, added the CIT to the Pull Request (& yes, as you already > said – this issue is being tracked via JIRA - PROTON-1171). > > Thanks a lot for the Wonderful Collaboration! > Sree > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: Thursday, April 7, 2016 3:52 AM > To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> > Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; > hm...@microsoft.com<mailto:hm...@microsoft.com> > Subject: Re: Proton-j Reactor - Receiver > > Hi Sree, > > Thanks for the analysis and PR, I'll try to take a proper look soon. > It's not an area of the code I'm familiar with so I'll need to have a > bit of a dig myself to see if the change seems ok. I'd note that any > not-insignificant bug fix such as this should probably have a test > with it (and a JIRA, though I see you have since created one of those) > :) > > Robbie > > On 6 April 2016 at 01:23, Garlapati Sreeram Kumar wrote: >> Hello Robbie, >> >> We are using proton-j client with SSL and many of our customers are hitting >> this issue. >> Here are my findings after debugging through this issue: >> >> - When incoming bytes arrive on the SocketChannel – proton-j client >> gets signaled by nio & as a result it unwinds the transport stack – as a >> result all the TransportInput implementations performs its task on the Read >> Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to >> frameparser etc). >> >> - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput >> reads(16k bytes) from _inputBuffer and the result - decoded bytes are >> written to _decodedInputBuffer – as an intermediate buffer. >> >> - It then flushes bytes from intermediate buffer to the next layer >> & invokes an _underlyingInput.Process() – to signal it that it has bytes in >> its input buffer. >> >> - If the underlyingInput (lets say FrameParser) buffer size is >> small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & >> Over time this accrues. >> >> The fix here is to flush decodedInputBuffer to the Next transport in the >> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer >> is empty. Here’s the pull request - >> https://github.com/apache/qpid-proton/pull/73 >> >> Pl. let me know if we need to do more to fix this issue comprehensively. >> >> Thx! >> Sree >> >> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> >> Sent: Thursday, March 31, 2016 9:19 AM >> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> >> Subject: Re: Proton-j Reactor - Receiver >> >> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar wrote: >>> Hello All! >>> >>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP >>> Messages (from Microsoft Azure Event Hubs): >>> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124 >>> >>> Am using the onDelivery(Event) callback to receive messages. I really >>> appreciate your help with this issue/behavior: >>> >>> ISSUE: I noticed that the last few messages on the Queue are not being >>> issued to onDelivery(Event) callback by the Reactor >>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and >>> discovered that the Transfer frames corresponding to those messages were >>> not even delivered to Client. Then, I looked at our Service Proton Frames >>> and can clearly s
Re: Proton-j Reactor - Receiver
Ah, excellent. I had actually started on testing this myself a little earlier, so I'll take a look and see whats what before continuing tomorrow. On taking an initial better look at things I think the change itself may need augmented to account for some other conditions too, need to investigate further to be sure. Robbie On 11 April 2016 at 17:37, Garlapati Sreeram Kumar wrote: > Thanks a lot for the Response Robbie! > Per your suggestion, added the CIT to the Pull Request (& yes, as you already > said – this issue is being tracked via JIRA - PROTON-1171). > > Thanks a lot for the Wonderful Collaboration! > Sree > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: Thursday, April 7, 2016 3:52 AM > To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> > Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; > hm...@microsoft.com<mailto:hm...@microsoft.com> > Subject: Re: Proton-j Reactor - Receiver > > Hi Sree, > > Thanks for the analysis and PR, I'll try to take a proper look soon. > It's not an area of the code I'm familiar with so I'll need to have a > bit of a dig myself to see if the change seems ok. I'd note that any > not-insignificant bug fix such as this should probably have a test > with it (and a JIRA, though I see you have since created one of those) > :) > > Robbie > > On 6 April 2016 at 01:23, Garlapati Sreeram Kumar wrote: >> Hello Robbie, >> >> We are using proton-j client with SSL and many of our customers are hitting >> this issue. >> Here are my findings after debugging through this issue: >> >> - When incoming bytes arrive on the SocketChannel – proton-j client >> gets signaled by nio & as a result it unwinds the transport stack – as a >> result all the TransportInput implementations performs its task on the Read >> Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to >> frameparser etc). >> >> - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput >> reads(16k bytes) from _inputBuffer and the result - decoded bytes are >> written to _decodedInputBuffer – as an intermediate buffer. >> >> - It then flushes bytes from intermediate buffer to the next layer >> & invokes an _underlyingInput.Process() – to signal it that it has bytes in >> its input buffer. >> >> - If the underlyingInput (lets say FrameParser) buffer size is >> small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & >> Over time this accrues. >> >> The fix here is to flush decodedInputBuffer to the Next transport in the >> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer >> is empty. Here’s the pull request - >> https://github.com/apache/qpid-proton/pull/73 >> >> Pl. let me know if we need to do more to fix this issue comprehensively. >> >> Thx! >> Sree >> >> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> >> Sent: Thursday, March 31, 2016 9:19 AM >> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> >> Subject: Re: Proton-j Reactor - Receiver >> >> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar wrote: >>> Hello All! >>> >>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP >>> Messages (from Microsoft Azure Event Hubs): >>> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124 >>> >>> Am using the onDelivery(Event) callback to receive messages. I really >>> appreciate your help with this issue/behavior: >>> >>> ISSUE: I noticed that the last few messages on the Queue are not being >>> issued to onDelivery(Event) callback by the Reactor >>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and >>> discovered that the Transfer frames corresponding to those messages were >>> not even delivered to Client. Then, I looked at our Service Proton Frames >>> and can clearly see that they are being delivered by the Service. And other >>> AMQP clients (for ex: .net client can see the Transfer frames) >>> - Is this a known behavior? >>> Does Reactor code path disable Nagle on underlying socket – could this be >>> related? or is there any other Configuration that we should be setting to >>> see all Transfer frames received on the Socket? >>> >>> Please advice. >>> >>> Thanks a lot in Advance! >>> Sree >>> >>> Sent from Mail for Windows 10 >>> >> >> I'm not aware of anyone else reporting anything like that. I don't see >> anything in the code suggesting the reactor sets TCP_NODELAY trueon >> the socket, but I wouldn't think that should matter here. >> >> The frame trace logging is done after the bytes are given to the >> Transport and are processed into frames, so a lack of logging could >> suggest various things such as they didnt actually get there, they >> werent processed, something went wrong before they did/were, something >> went wrong decoding them, etc. Its hard to say much more without more >> info. >> >> Robbie
RE: Proton-j Reactor - Receiver
Thanks a lot for the Response Robbie! Per your suggestion, added the CIT to the Pull Request (& yes, as you already said – this issue is being tracked via JIRA - PROTON-1171). Thanks a lot for the Wonderful Collaboration! Sree From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> Sent: Thursday, April 7, 2016 3:52 AM To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; hm...@microsoft.com<mailto:hm...@microsoft.com> Subject: Re: Proton-j Reactor - Receiver Hi Sree, Thanks for the analysis and PR, I'll try to take a proper look soon. It's not an area of the code I'm familiar with so I'll need to have a bit of a dig myself to see if the change seems ok. I'd note that any not-insignificant bug fix such as this should probably have a test with it (and a JIRA, though I see you have since created one of those) :) Robbie On 6 April 2016 at 01:23, Garlapati Sreeram Kumar wrote: > Hello Robbie, > > We are using proton-j client with SSL and many of our customers are hitting > this issue. > Here are my findings after debugging through this issue: > > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request - https://github.com/apache/qpid-proton/pull/73 > > Pl. let me know if we need to do more to fix this issue comprehensively. > > Thx! > Sree > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: Thursday, March 31, 2016 9:19 AM > To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> > Subject: Re: Proton-j Reactor - Receiver > > On 31 March 2016 at 04:32, Garlapati Sreeram Kumar wrote: >> Hello All! >> >> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages >> (from Microsoft Azure Event Hubs): >> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124 >> >> Am using the onDelivery(Event) callback to receive messages. I really >> appreciate your help with this issue/behavior: >> >> ISSUE: I noticed that the last few messages on the Queue are not being >> issued to onDelivery(Event) callback by the Reactor >> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and >> discovered that the Transfer frames corresponding to those messages were not >> even delivered to Client. Then, I looked at our Service Proton Frames and >> can clearly see that they are being delivered by the Service. And other AMQP >> clients (for ex: .net client can see the Transfer frames) >> - Is this a known behavior? >> Does Reactor code path disable Nagle on underlying socket – could this be >> related? or is there any other Configuration that we should be setting to >> see all Transfer frames received on the Socket? >> >> Please advice. >> >> Thanks a lot in Advance! >> Sree >> >> Sent from Mail for Windows 10 >> > > I'm not aware of anyone else reporting anything like that. I don't see > anything in the code suggesting the reactor sets TCP_NODELAY trueon > the socket, but I wouldn't think that should matter here. > > The frame trace logging is done after the bytes are given to the > Transport and are processed into frames, so a lack of logging could > suggest various things such as they didnt actually get there, they > werent processed, something went wrong before they did/were, something > went wrong decoding them, etc. Its hard to say much more without more > info. > > Robbie
Re: Proton-j Reactor - Receiver
Hi Sree, Thanks for the analysis and PR, I'll try to take a proper look soon. It's not an area of the code I'm familiar with so I'll need to have a bit of a dig myself to see if the change seems ok. I'd note that any not-insignificant bug fix such as this should probably have a test with it (and a JIRA, though I see you have since created one of those) :) Robbie On 6 April 2016 at 01:23, Garlapati Sreeram Kumar wrote: > Hello Robbie, > > We are using proton-j client with SSL and many of our customers are hitting > this issue. > Here are my findings after debugging through this issue: > > - When incoming bytes arrive on the SocketChannel – proton-j client > gets signaled by nio & as a result it unwinds the transport stack – as a > result all the TransportInput implementations performs its task on the Read > Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to > frameparser etc). > > - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput > reads(16k bytes) from _inputBuffer and the result - decoded bytes are written > to _decodedInputBuffer – as an intermediate buffer. > > - It then flushes bytes from intermediate buffer to the next layer & > invokes an _underlyingInput.Process() – to signal it that it has bytes in its > input buffer. > > - If the underlyingInput (lets say FrameParser) buffer size is small > – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over > time this accrues. > > The fix here is to flush decodedInputBuffer to the Next transport in the > Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is > empty. Here’s the pull request - https://github.com/apache/qpid-proton/pull/73 > > Pl. let me know if we need to do more to fix this issue comprehensively. > > Thx! > Sree > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: Thursday, March 31, 2016 9:19 AM > To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> > Subject: Re: Proton-j Reactor - Receiver > > On 31 March 2016 at 04:32, Garlapati Sreeram Kumar wrote: >> Hello All! >> >> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages >> (from Microsoft Azure Event Hubs): >> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124 >> >> Am using the onDelivery(Event) callback to receive messages. I really >> appreciate your help with this issue/behavior: >> >> ISSUE: I noticed that the last few messages on the Queue are not being >> issued to onDelivery(Event) callback by the Reactor >> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and >> discovered that the Transfer frames corresponding to those messages were not >> even delivered to Client. Then, I looked at our Service Proton Frames and >> can clearly see that they are being delivered by the Service. And other AMQP >> clients (for ex: .net client can see the Transfer frames) >> - Is this a known behavior? >> Does Reactor code path disable Nagle on underlying socket – could this be >> related? or is there any other Configuration that we should be setting to >> see all Transfer frames received on the Socket? >> >> Please advice. >> >> Thanks a lot in Advance! >> Sree >> >> Sent from Mail for Windows 10 >> > > I'm not aware of anyone else reporting anything like that. I don't see > anything in the code suggesting the reactor sets TCP_NODELAY trueon > the socket, but I wouldn't think that should matter here. > > The frame trace logging is done after the bytes are given to the > Transport and are processed into frames, so a lack of logging could > suggest various things such as they didnt actually get there, they > werent processed, something went wrong before they did/were, something > went wrong decoding them, etc. Its hard to say much more without more > info. > > Robbie
RE: Proton-j Reactor - Receiver
Hello Robbie, We are using proton-j client with SSL and many of our customers are hitting this issue. Here are my findings after debugging through this issue: - When incoming bytes arrive on the SocketChannel – proton-j client gets signaled by nio & as a result it unwinds the transport stack – as a result all the TransportInput implementations performs its task on the Read Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to frameparser etc). - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput reads(16k bytes) from _inputBuffer and the result - decoded bytes are written to _decodedInputBuffer – as an intermediate buffer. - It then flushes bytes from intermediate buffer to the next layer & invokes an _underlyingInput.Process() – to signal it that it has bytes in its input buffer. - If the underlyingInput (lets say FrameParser) buffer size is small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over time this accrues. The fix here is to flush decodedInputBuffer to the Next transport in the Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is empty. Here’s the pull request - https://github.com/apache/qpid-proton/pull/73 Pl. let me know if we need to do more to fix this issue comprehensively. Thx! Sree From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> Sent: Thursday, March 31, 2016 9:19 AM To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> Subject: Re: Proton-j Reactor - Receiver On 31 March 2016 at 04:32, Garlapati Sreeram Kumar wrote: > Hello All! > > I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages > (from Microsoft Azure Event Hubs): > https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124 > > Am using the onDelivery(Event) callback to receive messages. I really > appreciate your help with this issue/behavior: > > ISSUE: I noticed that the last few messages on the Queue are not being issued > to onDelivery(Event) callback by the Reactor > - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and > discovered that the Transfer frames corresponding to those messages were not > even delivered to Client. Then, I looked at our Service Proton Frames and can > clearly see that they are being delivered by the Service. And other AMQP > clients (for ex: .net client can see the Transfer frames) > - Is this a known behavior? > Does Reactor code path disable Nagle on underlying socket – could this be > related? or is there any other Configuration that we should be setting to see > all Transfer frames received on the Socket? > > Please advice. > > Thanks a lot in Advance! > Sree > > Sent from Mail for Windows 10 > I'm not aware of anyone else reporting anything like that. I don't see anything in the code suggesting the reactor sets TCP_NODELAY trueon the socket, but I wouldn't think that should matter here. The frame trace logging is done after the bytes are given to the Transport and are processed into frames, so a lack of logging could suggest various things such as they didnt actually get there, they werent processed, something went wrong before they did/were, something went wrong decoding them, etc. Its hard to say much more without more info. Robbie
Re: Proton-j Reactor - Receiver
On 31 March 2016 at 04:32, Garlapati Sreeram Kumar wrote: > Hello All! > > I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages > (from Microsoft Azure Event Hubs): > https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124 > > Am using the onDelivery(Event) callback to receive messages. I really > appreciate your help with this issue/behavior: > > ISSUE: I noticed that the last few messages on the Queue are not being issued > to onDelivery(Event) callback by the Reactor > - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and > discovered that the Transfer frames corresponding to those messages were not > even delivered to Client. Then, I looked at our Service Proton Frames and can > clearly see that they are being delivered by the Service. And other AMQP > clients (for ex: .net client can see the Transfer frames) > - Is this a known behavior? > Does Reactor code path disable Nagle on underlying socket – could this be > related? or is there any other Configuration that we should be setting to see > all Transfer frames received on the Socket? > > Please advice. > > Thanks a lot in Advance! > Sree > > Sent from Mail for Windows 10 > I'm not aware of anyone else reporting anything like that. I don't see anything in the code suggesting the reactor sets TCP_NODELAY trueon the socket, but I wouldn't think that should matter here. The frame trace logging is done after the bytes are given to the Transport and are processed into frames, so a lack of logging could suggest various things such as they didnt actually get there, they werent processed, something went wrong before they did/were, something went wrong decoding them, etc. Its hard to say much more without more info. Robbie
Proton-j Reactor - Receiver
Hello All! I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages (from Microsoft Azure Event Hubs): https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124 Am using the onDelivery(Event) callback to receive messages. I really appreciate your help with this issue/behavior: ISSUE: I noticed that the last few messages on the Queue are not being issued to onDelivery(Event) callback by the Reactor - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and discovered that the Transfer frames corresponding to those messages were not even delivered to Client. Then, I looked at our Service Proton Frames and can clearly see that they are being delivered by the Service. And other AMQP clients (for ex: .net client can see the Transfer frames) - Is this a known behavior? Does Reactor code path disable Nagle on underlying socket – could this be related? or is there any other Configuration that we should be setting to see all Transfer frames received on the Socket? Please advice. Thanks a lot in Advance! Sree Sent from Mail for Windows 10