RE: Proton-j Reactor - Receiver

[Garlapati Sreeram Kumar]

Hello Robbie – That’s correct, please go ahead with the release 0.12.2.
As per your suggestion, I will follow up on this issue with a Simple reproducer.

Thx!
Sree

From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
Sent: Friday, April 15, 2016 7:09 AM
To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; 
hm...@microsoft.com<mailto:hm...@microsoft.com>
Subject: Re: Proton-j Reactor - Receiver

Hi Sree,

Your attachment didnt make it, the mailing list strips almost all
attachments, particularly of any non trivial size. Best to attach
stuff to JIRAs or provide links to them elsewhere.

I had another look at it and still don't see a way for it to do what
it did before, I'd guess it may be something slightly different. I
think I'll need more analysis and/or a reproducer from you to make any
further progress.

0.12.2 currently has the votes to pass. If it stays that way I
probably lean towards proceeding with its release unless more concrete
details becomes obvious, since it does seem to give a good improvement
over the previous behaviour.

Robbie

On 15 April 2016 at 01:35, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
> Hello Robbie,
>
>
>
> After couple hours of Stress Run with the proton-j change – we still could
> repro the Receive Stuck problem. Although – the below bug fix places us in a
> much better state now.
>
> Attached a screenshot of Objects sizes on Heap – which corresponds to the
> codepath that you fixed (SimpleSslTransportWrapper).
>
> Please see if this rings any bells – I am more than happy to share more
> details (proton traces & dump – pls suggest if you will need more details).
>
>
>
> Thx!
>
> Sree
>
>
>
> From: Garlapati Sreeram Kumar
> Sent: Monday, April 11, 2016 11:50 AM
> To: Robbie Gemmell; proton@qpid.apache.org
> Cc: SeongJoon Kwak (SJ); hm...@microsoft.com
> Subject: RE: Proton-j Reactor - Receiver
>
>
>
> Awesome.
>
> To make it easy - added you as collaborator to my fork of Proton & here’s
> the branch from which I submitted the PR:
> https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck
>
> Thx!
> Sree
>
> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
> Sent: Monday, April 11, 2016 9:52 AM
> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>;
> hm...@microsoft.com<mailto:hm...@microsoft.com>
> Subject: Re: Proton-j Reactor - Receiver
>
> Ah, excellent. I had actually started on testing this myself a little
> earlier, so I'll take a look and see whats what before continuing
> tomorrow. On taking an initial better look at things I think the
> change itself may need augmented to account for some other conditions
> too, need to investigate further to be sure.
>
> Robbie
>
> On 11 April 2016 at 17:37, Garlapati Sreeram Kumar <sreer...@live.com>
> wrote:
>> Thanks a lot for the Response Robbie!
>> Per your suggestion, added the CIT to the Pull Request (& yes, as you
>> already said – this issue is being tracked via JIRA - PROTON-1171).
>>
>> Thanks a lot for the Wonderful Collaboration!
>> Sree
>>
>> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
>> Sent: Thursday, April 7, 2016 3:52 AM
>> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
>> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>;
>> hm...@microsoft.com<mailto:hm...@microsoft.com>
>> Subject: Re: Proton-j Reactor - Receiver
>>
>> Hi Sree,
>>
>> Thanks for the analysis and PR, I'll try to take a proper look soon.
>> It's not an area of the code I'm familiar with so I'll need to have a
>> bit of a dig myself to see if the change seems ok. I'd note that any
>> not-insignificant bug fix such as this should probably have a test
>> with it (and a JIRA, though I see you have since created one of those)
>> :)
>>
>> Robbie
>>
>> On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sreer...@live.com>
>> wrote:
>>> Hello Robbie,
>>>
>>> We are using proton-j client with SSL and many of our customers are
>>> hitting this issue.
>>> Here are my findings after debugging through this issue:
>>>
>>> -  When incoming bytes arrive on the SocketChannel – proton-j
>>> client gets signaled by nio & as a result it unwinds the transport stack –
>>> as a result all the TransportInput implementations performs its task on the
>>> Read Bytes and hands off to the Next Layer in the stack (transport to ssl,
>>> ssl to frameparser etc).
>>

RE: Proton-j Reactor - Receiver

[Garlapati Sreeram Kumar]

Hello Robbie,

After couple hours of Stress Run with the proton-j change – we still could 
repro the Receive Stuck problem. Although – the below bug fix places us in a 
much better state now.
Attached a screenshot of Objects sizes on Heap – which corresponds to the 
codepath that you fixed (SimpleSslTransportWrapper).
Please see if this rings any bells – I am more than happy to share more details 
(proton traces & dump – pls suggest if you will need more details).

Thx!
Sree

From: Garlapati Sreeram Kumar<mailto:sreer...@live.com>
Sent: Monday, April 11, 2016 11:50 AM
To: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>; 
proton@qpid.apache.org<mailto:proton@qpid.apache.org>
Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; 
hm...@microsoft.com<mailto:hm...@microsoft.com>
Subject: RE: Proton-j Reactor - Receiver

Awesome.

To make it easy - added you as collaborator to my fork of Proton & here’s the 
branch from which I submitted the PR: 
https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck

Thx!
Sree

From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
Sent: Monday, April 11, 2016 9:52 AM
To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; 
hm...@microsoft.com<mailto:hm...@microsoft.com>
Subject: Re: Proton-j Reactor - Receiver

Ah, excellent. I had actually started on testing this myself a little
earlier, so I'll take a look and see whats what before continuing
tomorrow. On taking an initial better look at things I think the
change itself may need augmented to account for some other conditions
too, need to investigate further to be sure.

Robbie

On 11 April 2016 at 17:37, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
> Thanks a lot for the Response Robbie!
> Per your suggestion, added the CIT to the Pull Request (& yes, as you already 
> said – this issue is being tracked via JIRA - PROTON-1171).
>
> Thanks a lot for the Wonderful Collaboration!
> Sree
>
> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
> Sent: Thursday, April 7, 2016 3:52 AM
> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; 
> hm...@microsoft.com<mailto:hm...@microsoft.com>
> Subject: Re: Proton-j Reactor - Receiver
>
> Hi Sree,
>
> Thanks for the analysis and PR, I'll try to take a proper look soon.
> It's not an area of the code I'm familiar with so I'll need to have a
> bit of a dig myself to see if the change seems ok. I'd note that any
> not-insignificant bug fix such as this should probably have a test
> with it (and a JIRA, though I see you have since created one of those)
> :)
>
> Robbie
>
> On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
>> Hello Robbie,
>>
>> We are using proton-j client with SSL and many of our customers are hitting 
>> this issue.
>> Here are my findings after debugging through this issue:
>>
>> -  When incoming bytes arrive on the SocketChannel – proton-j client 
>> gets signaled by nio & as a result it unwinds the transport stack – as a 
>> result all the TransportInput implementations performs its task on the Read 
>> Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to 
>> frameparser etc).
>>
>> -  While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput 
>> reads(16k bytes) from _inputBuffer and the result - decoded bytes are 
>> written to _decodedInputBuffer – as an intermediate buffer.
>>
>> -  It then flushes bytes from intermediate buffer to the next layer 
>> & invokes an _underlyingInput.Process() – to signal it that it has bytes in 
>> its input buffer.
>>
>> -  If the underlyingInput (lets say FrameParser) buffer size is 
>> small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & 
>> Over time this accrues.
>>
>> The fix here is to flush decodedInputBuffer to the Next transport in the 
>> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer 
>> is empty. Here’s the pull request - 
>> https://github.com/apache/qpid-proton/pull/73
>>
>> Pl. let me know if we need to do more to fix this issue comprehensively.
>>
>> Thx!
>> Sree
>>
>> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
>> Sent: Thursday, March 31, 2016 9:19 AM
>> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
>> Subject: Re: Proton-j Reactor - Receiver
>>
>> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
>>> Hello All!
>>>
&g

RE: Proton-j Reactor - Receiver

[Garlapati Sreeram Kumar]

Awesome.

To make it easy - added you as collaborator to my fork of Proton & here’s the 
branch from which I submitted the PR: 
https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck

Thx!
Sree

From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
Sent: Monday, April 11, 2016 9:52 AM
To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; 
hm...@microsoft.com<mailto:hm...@microsoft.com>
Subject: Re: Proton-j Reactor - Receiver

Ah, excellent. I had actually started on testing this myself a little
earlier, so I'll take a look and see whats what before continuing
tomorrow. On taking an initial better look at things I think the
change itself may need augmented to account for some other conditions
too, need to investigate further to be sure.

Robbie

On 11 April 2016 at 17:37, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
> Thanks a lot for the Response Robbie!
> Per your suggestion, added the CIT to the Pull Request (& yes, as you already 
> said – this issue is being tracked via JIRA - PROTON-1171).
>
> Thanks a lot for the Wonderful Collaboration!
> Sree
>
> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
> Sent: Thursday, April 7, 2016 3:52 AM
> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; 
> hm...@microsoft.com<mailto:hm...@microsoft.com>
> Subject: Re: Proton-j Reactor - Receiver
>
> Hi Sree,
>
> Thanks for the analysis and PR, I'll try to take a proper look soon.
> It's not an area of the code I'm familiar with so I'll need to have a
> bit of a dig myself to see if the change seems ok. I'd note that any
> not-insignificant bug fix such as this should probably have a test
> with it (and a JIRA, though I see you have since created one of those)
> :)
>
> Robbie
>
> On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
>> Hello Robbie,
>>
>> We are using proton-j client with SSL and many of our customers are hitting 
>> this issue.
>> Here are my findings after debugging through this issue:
>>
>> -  When incoming bytes arrive on the SocketChannel – proton-j client 
>> gets signaled by nio & as a result it unwinds the transport stack – as a 
>> result all the TransportInput implementations performs its task on the Read 
>> Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to 
>> frameparser etc).
>>
>> -  While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput 
>> reads(16k bytes) from _inputBuffer and the result - decoded bytes are 
>> written to _decodedInputBuffer – as an intermediate buffer.
>>
>> -  It then flushes bytes from intermediate buffer to the next layer 
>> & invokes an _underlyingInput.Process() – to signal it that it has bytes in 
>> its input buffer.
>>
>> -  If the underlyingInput (lets say FrameParser) buffer size is 
>> small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & 
>> Over time this accrues.
>>
>> The fix here is to flush decodedInputBuffer to the Next transport in the 
>> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer 
>> is empty. Here’s the pull request - 
>> https://github.com/apache/qpid-proton/pull/73
>>
>> Pl. let me know if we need to do more to fix this issue comprehensively.
>>
>> Thx!
>> Sree
>>
>> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
>> Sent: Thursday, March 31, 2016 9:19 AM
>> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
>> Subject: Re: Proton-j Reactor - Receiver
>>
>> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
>>> Hello All!
>>>
>>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP 
>>> Messages (from Microsoft Azure Event Hubs): 
>>> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>>>
>>> Am using the onDelivery(Event) callback to receive messages. I really 
>>> appreciate your help with this issue/behavior:
>>>
>>> ISSUE: I noticed that the last few messages on the Queue are not being 
>>> issued to onDelivery(Event) callback by the Reactor
>>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and 
>>> discovered that the Transfer frames corresponding to those messages were 
>>> not even delivered to Client. Then, I looked at our Service 

Re: Proton-j Reactor - Receiver

[Robbie Gemmell]

Ah, excellent. I had actually started on testing this myself a little
earlier, so I'll take a look and see whats what before continuing
tomorrow. On taking an initial better look at things I think the
change itself may need augmented to account for some other conditions
too, need to investigate further to be sure.

Robbie

On 11 April 2016 at 17:37, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
> Thanks a lot for the Response Robbie!
> Per your suggestion, added the CIT to the Pull Request (& yes, as you already 
> said – this issue is being tracked via JIRA - PROTON-1171).
>
> Thanks a lot for the Wonderful Collaboration!
> Sree
>
> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
> Sent: Thursday, April 7, 2016 3:52 AM
> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; 
> hm...@microsoft.com<mailto:hm...@microsoft.com>
> Subject: Re: Proton-j Reactor - Receiver
>
> Hi Sree,
>
> Thanks for the analysis and PR, I'll try to take a proper look soon.
> It's not an area of the code I'm familiar with so I'll need to have a
> bit of a dig myself to see if the change seems ok. I'd note that any
> not-insignificant bug fix such as this should probably have a test
> with it (and a JIRA, though I see you have since created one of those)
> :)
>
> Robbie
>
> On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
>> Hello Robbie,
>>
>> We are using proton-j client with SSL and many of our customers are hitting 
>> this issue.
>> Here are my findings after debugging through this issue:
>>
>> -  When incoming bytes arrive on the SocketChannel – proton-j client 
>> gets signaled by nio & as a result it unwinds the transport stack – as a 
>> result all the TransportInput implementations performs its task on the Read 
>> Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to 
>> frameparser etc).
>>
>> -  While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput 
>> reads(16k bytes) from _inputBuffer and the result - decoded bytes are 
>> written to _decodedInputBuffer – as an intermediate buffer.
>>
>> -  It then flushes bytes from intermediate buffer to the next layer 
>> & invokes an _underlyingInput.Process() – to signal it that it has bytes in 
>> its input buffer.
>>
>> -  If the underlyingInput (lets say FrameParser) buffer size is 
>> small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & 
>> Over time this accrues.
>>
>> The fix here is to flush decodedInputBuffer to the Next transport in the 
>> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer 
>> is empty. Here’s the pull request - 
>> https://github.com/apache/qpid-proton/pull/73
>>
>> Pl. let me know if we need to do more to fix this issue comprehensively.
>>
>> Thx!
>> Sree
>>
>> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
>> Sent: Thursday, March 31, 2016 9:19 AM
>> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
>> Subject: Re: Proton-j Reactor - Receiver
>>
>> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
>>> Hello All!
>>>
>>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP 
>>> Messages (from Microsoft Azure Event Hubs): 
>>> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>>>
>>> Am using the onDelivery(Event) callback to receive messages. I really 
>>> appreciate your help with this issue/behavior:
>>>
>>> ISSUE: I noticed that the last few messages on the Queue are not being 
>>> issued to onDelivery(Event) callback by the Reactor
>>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and 
>>> discovered that the Transfer frames corresponding to those messages were 
>>> not even delivered to Client. Then, I looked at our Service Proton Frames 
>>> and can clearly see that they are being delivered by the Service. And other 
>>> AMQP clients (for ex: .net client can see the Transfer frames)
>>> - Is this a known behavior?
>>> Does Reactor code path disable Nagle on underlying socket – could this be 
>>> related? or is there any other Configuration that we should be setting to 
>>> see all Transfer frames received on the Socket?
>>>
>>> Please advice.
>>>
>>> 

RE: Proton-j Reactor - Receiver

[Garlapati Sreeram Kumar]

Thanks a lot for the Response Robbie!
Per your suggestion, added the CIT to the Pull Request (& yes, as you already 
said – this issue is being tracked via JIRA - PROTON-1171).

Thanks a lot for the Wonderful Collaboration!
Sree

From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
Sent: Thursday, April 7, 2016 3:52 AM
To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; 
hm...@microsoft.com<mailto:hm...@microsoft.com>
Subject: Re: Proton-j Reactor - Receiver

Hi Sree,

Thanks for the analysis and PR, I'll try to take a proper look soon.
It's not an area of the code I'm familiar with so I'll need to have a
bit of a dig myself to see if the change seems ok. I'd note that any
not-insignificant bug fix such as this should probably have a test
with it (and a JIRA, though I see you have since created one of those)
:)

Robbie

On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
> Hello Robbie,
>
> We are using proton-j client with SSL and many of our customers are hitting 
> this issue.
> Here are my findings after debugging through this issue:
>
> -  When incoming bytes arrive on the SocketChannel – proton-j client 
> gets signaled by nio & as a result it unwinds the transport stack – as a 
> result all the TransportInput implementations performs its task on the Read 
> Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to 
> frameparser etc).
>
> -  While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput 
> reads(16k bytes) from _inputBuffer and the result - decoded bytes are written 
> to _decodedInputBuffer – as an intermediate buffer.
>
> -  It then flushes bytes from intermediate buffer to the next layer & 
> invokes an _underlyingInput.Process() – to signal it that it has bytes in its 
> input buffer.
>
> -  If the underlyingInput (lets say FrameParser) buffer size is small 
> – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over 
> time this accrues.
>
> The fix here is to flush decodedInputBuffer to the Next transport in the 
> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is 
> empty. Here’s the pull request - https://github.com/apache/qpid-proton/pull/73
>
> Pl. let me know if we need to do more to fix this issue comprehensively.
>
> Thx!
> Sree
>
> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
> Sent: Thursday, March 31, 2016 9:19 AM
> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
> Subject: Re: Proton-j Reactor - Receiver
>
> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
>> Hello All!
>>
>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages 
>> (from Microsoft Azure Event Hubs): 
>> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>>
>> Am using the onDelivery(Event) callback to receive messages. I really 
>> appreciate your help with this issue/behavior:
>>
>> ISSUE: I noticed that the last few messages on the Queue are not being 
>> issued to onDelivery(Event) callback by the Reactor
>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and 
>> discovered that the Transfer frames corresponding to those messages were not 
>> even delivered to Client. Then, I looked at our Service Proton Frames and 
>> can clearly see that they are being delivered by the Service. And other AMQP 
>> clients (for ex: .net client can see the Transfer frames)
>> - Is this a known behavior?
>> Does Reactor code path disable Nagle on underlying socket – could this be 
>> related? or is there any other Configuration that we should be setting to 
>> see all Transfer frames received on the Socket?
>>
>> Please advice.
>>
>> Thanks a lot in Advance!
>> Sree
>>
>> Sent from Mail for Windows 10
>>
>
> I'm not aware of anyone else reporting anything like that. I don't see
> anything in the code suggesting the reactor sets TCP_NODELAY trueon
> the socket, but I wouldn't think that should matter here.
>
> The frame trace logging is done after the bytes are given to the
> Transport and are processed into frames, so a lack of logging could
> suggest various things such as they didnt actually get there, they
> werent processed, something went wrong before they did/were, something
> went wrong decoding them, etc. Its hard to say much more without more
> info.
>
> Robbie

Re: Proton-j Reactor - Receiver

[Robbie Gemmell]

Hi Sree,

Thanks for the analysis and PR, I'll try to take a proper look soon.
It's not an area of the code I'm familiar with so I'll need to have a
bit of a dig myself to see if the change seems ok. I'd note that any
not-insignificant bug fix such as this should probably have a test
with it (and a JIRA, though I see you have since created one of those)
:)

Robbie

On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
> Hello Robbie,
>
> We are using proton-j client with SSL and many of our customers are hitting 
> this issue.
> Here are my findings after debugging through this issue:
>
> -  When incoming bytes arrive on the SocketChannel – proton-j client 
> gets signaled by nio & as a result it unwinds the transport stack – as a 
> result all the TransportInput implementations performs its task on the Read 
> Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to 
> frameparser etc).
>
> -  While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput 
> reads(16k bytes) from _inputBuffer and the result - decoded bytes are written 
> to _decodedInputBuffer – as an intermediate buffer.
>
> -  It then flushes bytes from intermediate buffer to the next layer & 
> invokes an _underlyingInput.Process() – to signal it that it has bytes in its 
> input buffer.
>
> -  If the underlyingInput (lets say FrameParser) buffer size is small 
> – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over 
> time this accrues.
>
> The fix here is to flush decodedInputBuffer to the Next transport in the 
> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is 
> empty. Here’s the pull request - https://github.com/apache/qpid-proton/pull/73
>
> Pl. let me know if we need to do more to fix this issue comprehensively.
>
> Thx!
> Sree
>
> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
> Sent: Thursday, March 31, 2016 9:19 AM
> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
> Subject: Re: Proton-j Reactor - Receiver
>
> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
>> Hello All!
>>
>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages 
>> (from Microsoft Azure Event Hubs): 
>> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>>
>> Am using the onDelivery(Event) callback to receive messages. I really 
>> appreciate your help with this issue/behavior:
>>
>> ISSUE: I noticed that the last few messages on the Queue are not being 
>> issued to onDelivery(Event) callback by the Reactor
>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and 
>> discovered that the Transfer frames corresponding to those messages were not 
>> even delivered to Client. Then, I looked at our Service Proton Frames and 
>> can clearly see that they are being delivered by the Service. And other AMQP 
>> clients (for ex: .net client can see the Transfer frames)
>> - Is this a known behavior?
>> Does Reactor code path disable Nagle on underlying socket – could this be 
>> related? or is there any other Configuration that we should be setting to 
>> see all Transfer frames received on the Socket?
>>
>> Please advice.
>>
>> Thanks a lot in Advance!
>> Sree
>>
>> Sent from Mail for Windows 10
>>
>
> I'm not aware of anyone else reporting anything like that. I don't see
> anything in the code suggesting the reactor sets TCP_NODELAY trueon
> the socket, but I wouldn't think that should matter here.
>
> The frame trace logging is done after the bytes are given to the
> Transport and are processed into frames, so a lack of logging could
> suggest various things such as they didnt actually get there, they
> werent processed, something went wrong before they did/were, something
> went wrong decoding them, etc. Its hard to say much more without more
> info.
>
> Robbie

RE: Proton-j Reactor - Receiver

[Garlapati Sreeram Kumar]

Hello Robbie,

We are using proton-j client with SSL and many of our customers are hitting 
this issue.
Here are my findings after debugging through this issue:

-  When incoming bytes arrive on the SocketChannel – proton-j client 
gets signaled by nio & as a result it unwinds the transport stack – as a result 
all the TransportInput implementations performs its task on the Read Bytes and 
hands off to the Next Layer in the stack (transport to ssl, ssl to frameparser 
etc).

-  While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput 
reads(16k bytes) from _inputBuffer and the result - decoded bytes are written 
to _decodedInputBuffer – as an intermediate buffer.

-  It then flushes bytes from intermediate buffer to the next layer & 
invokes an _underlyingInput.Process() – to signal it that it has bytes in its 
input buffer.

-  If the underlyingInput (lets say FrameParser) buffer size is small – 
lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over time 
this accrues.

The fix here is to flush decodedInputBuffer to the Next transport in the 
Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is 
empty. Here’s the pull request - https://github.com/apache/qpid-proton/pull/73

Pl. let me know if we need to do more to fix this issue comprehensively.

Thx!
Sree

From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com>
Sent: Thursday, March 31, 2016 9:19 AM
To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
Subject: Re: Proton-j Reactor - Receiver

On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sreer...@live.com> wrote:
> Hello All!
>
> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages 
> (from Microsoft Azure Event Hubs): 
> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>
> Am using the onDelivery(Event) callback to receive messages. I really 
> appreciate your help with this issue/behavior:
>
> ISSUE: I noticed that the last few messages on the Queue are not being issued 
> to onDelivery(Event) callback by the Reactor
> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and 
> discovered that the Transfer frames corresponding to those messages were not 
> even delivered to Client. Then, I looked at our Service Proton Frames and can 
> clearly see that they are being delivered by the Service. And other AMQP 
> clients (for ex: .net client can see the Transfer frames)
> - Is this a known behavior?
> Does Reactor code path disable Nagle on underlying socket – could this be 
> related? or is there any other Configuration that we should be setting to see 
> all Transfer frames received on the Socket?
>
> Please advice.
>
> Thanks a lot in Advance!
> Sree
>
> Sent from Mail for Windows 10
>

I'm not aware of anyone else reporting anything like that. I don't see
anything in the code suggesting the reactor sets TCP_NODELAY trueon
the socket, but I wouldn't think that should matter here.

The frame trace logging is done after the bytes are given to the
Transport and are processed into frames, so a lack of logging could
suggest various things such as they didnt actually get there, they
werent processed, something went wrong before they did/were, something
went wrong decoding them, etc. Its hard to say much more without more
info.

Robbie

Re: Proton-j Reactor - Receiver

[Robbie Gemmell]

On 31 March 2016 at 04:32, Garlapati Sreeram Kumar  wrote:
> Hello All!
>
> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages 
> (from Microsoft Azure Event Hubs): 
> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>
> Am using the onDelivery(Event) callback to receive messages. I really 
> appreciate your help with this issue/behavior:
>
> ISSUE: I noticed that the last few messages on the Queue are not being issued 
> to onDelivery(Event) callback by the Reactor
> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and 
> discovered that the Transfer frames corresponding to those messages were not 
> even delivered to Client. Then, I looked at our Service Proton Frames and can 
> clearly see that they are being delivered by the Service. And other AMQP 
> clients (for ex: .net client can see the Transfer frames)
> - Is this a known behavior?
> Does Reactor code path disable Nagle on underlying socket – could this be 
> related? or is there any other Configuration that we should be setting to see 
> all Transfer frames received on the Socket?
>
> Please advice.
>
> Thanks a lot in Advance!
> Sree
>
> Sent from Mail for Windows 10
>

I'm not aware of anyone else reporting anything like that. I don't see
anything in the code suggesting the reactor sets TCP_NODELAY trueon
the socket, but I wouldn't think that should matter here.

The frame trace logging is done after the bytes are given to the
Transport and are processed into frames, so a lack of logging could
suggest various things such as they didnt actually get there, they
werent processed, something went wrong before they did/were, something
went wrong decoding them, etc. Its hard to say much more without more
info.

Robbie