Re: AMQP 1.0 session outgoing-window usage / meaning
On 07/08/2015 02:22 AM, Rafael Schloming wrote: a value of zero is actually what signals that the receiver needs to take some action here, and arguably an initial value of zero is correct since it is signaling that the receiver needs to take action (in this case issue credit). My interpretation is that if 0 is sent as the initial value, the sender cannot legally send any transfers without first expanding the window by sending a flow with a non-zero value. Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). In the case where the sender's implementation involves a fixed amount of buffer space and requires messages to be settled before it can send more, the receiver would not be able to know that without getting some signal. So to my mind that is the only case for which it would make sense to send an outgoing window of 0. (I'm not sure how useful this is in practice and I don't believe it applies to proton at present anyway). I think as it stands proton is violating the spec, and should be changed to send a non-zero outgoing window.
Re: AMQP 1.0 session outgoing-window usage / meaning
On 8 July 2015 at 10:03, Gordon Sim g...@redhat.com wrote: On 07/08/2015 02:22 AM, Rafael Schloming wrote: a value of zero is actually what signals that the receiver needs to take some action here, and arguably an initial value of zero is correct since it is signaling that the receiver needs to take action (in this case issue credit). My interpretation is that if 0 is sent as the initial value, the sender cannot legally send any transfers without first expanding the window by sending a flow with a non-zero value. Agreed. We don't currently do that, so we are typically violating the window (Messenger being an exception, on its initial send to a node at least). In the case the remote incoming window is also 0 (which it is in the case which prompted this discussion) we would also have to synchronously wait for a flow 'response' increasing it before we could send anything, since we also need to know the remote incoming window actually enabled us to send. Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). That was my take on reading things, that the two are essentially separate mechanisms, and on top the two windows are separate as well. If the initial outgoing window was set to 0 because the reciever hasnt issued, would there be a need for the field to exist on Begin? It would seem like much of the time it wouldnt be used and a Flow would have to be used instead. In the case where the sender's implementation involves a fixed amount of buffer space and requires messages to be settled before it can send more, the receiver would not be able to know that without getting some signal. So to my mind that is the only case for which it would make sense to send an outgoing window of 0. (I'm not sure how useful this is in practice and I don't believe it applies to proton at present anyway). I think as it stands proton is violating the spec, and should be changed to send a non-zero outgoing window. That is what I did for the proposed changes on https://issues.apache.org/jira/browse/PROTON-936 / https://github.com/apache/qpid-proton/pull/42. Essentially setting it initially on begin to max int (unless otherwise configured) and leaving it there for any subsequent flows. We could later (when we arent right before a release) look to make it smarter if needed. Robbie
Re: AMQP 1.0 session outgoing-window usage / meaning
// quotes from the spec --- 2.5.6 Session Flow Control The Session Endpoint assigns each outgoing transfer frame an implicit transfer-id from a session scoped sequence. outgoing-window: The outgoing-window defines the maximum number of outgoing transfer frames that the endpoint can currently send. This identifies a current maximum outgoing transfer-id that can be computed by subtracting one from the sum of outgoing-window and next-outgoing-id. // end quotes --- Outgoing-window is a mandatory field on flow. The sender's outgoing window is communicated to the peer. If the sender sends a 0 outgoing window but subsequently sends a transfer frame, the implicit outgoing transfer-id would exceed the limit implied by the flow frame. I agree that the sender can maintain a local outgoing window to satisfy any local implementation specific logic, but the value communicated to the peer must allow further transfer frames to be sent. I don't think it makes senses to use 0 outgoing window to request for credit (assuming it is the link credit mentioned in early mails). Using a 0 link credit would make more sense. On Wed, Jul 8, 2015 at 8:03 AM, Rafael Schloming r...@alum.mit.edu wrote: On Wed, Jul 8, 2015 at 2:38 AM, Robbie Gemmell robbie.gemm...@gmail.com wrote: On 8 July 2015 at 10:03, Gordon Sim g...@redhat.com wrote: On 07/08/2015 02:22 AM, Rafael Schloming wrote: a value of zero is actually what signals that the receiver needs to take some action here, and arguably an initial value of zero is correct since it is signaling that the receiver needs to take action (in this case issue credit). My interpretation is that if 0 is sent as the initial value, the sender cannot legally send any transfers without first expanding the window by sending a flow with a non-zero value. Agreed. We don't currently do that, so we are typically violating the window (Messenger being an exception, on its initial send to a node at least). In the case the remote incoming window is also 0 (which it is in the case which prompted this discussion) we would also have to synchronously wait for a flow 'response' increasing it before we could send anything, since we also need to know the remote incoming window actually enabled us to send. See my reply to Gordon for more details, but I think we need to be careful in distinguishing between the rules the spec mandates for updating/maintaining the value locally, and the rules the spec mandates for when it is necessary to communicate the value to the peer. These definitely should not be conflated because you don't want to notify the peer whenever you update the value locally, this would be way to chatty. Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). That was my take on reading things, that the two are essentially separate mechanisms, and on top the two windows are separate as well. If the initial outgoing window was set to 0 because the reciever hasnt issued, would there be a need for the field to exist on Begin? It would seem like much of the time it wouldnt be used and a Flow would have to be used instead. In the case where the sender's implementation involves a fixed amount of buffer space and requires messages to be settled before it can send more, the receiver would not be able to know that without getting some signal. So to my mind that is the only case for which it would make sense to send an outgoing window of 0. (I'm not sure how useful this is in practice and I don't believe it applies to proton at present anyway). I think as it stands proton is violating the spec, and should be changed to send a non-zero outgoing window. That is what I did for the proposed changes on https://issues.apache.org/jira/browse/PROTON-936 / https://github.com/apache/qpid-proton/pull/42. Essentially setting it initially on begin to max int (unless otherwise configured) and leaving it there for any subsequent flows. We could later (when we arent right before a release) look to make it smarter if needed. See my other email for more details, but I agree this seems like a reasonable change for now. --Rafael
Re: AMQP 1.0 session outgoing-window usage / meaning
On Wed, Jul 8, 2015 at 2:38 AM, Robbie Gemmell robbie.gemm...@gmail.com wrote: On 8 July 2015 at 10:03, Gordon Sim g...@redhat.com wrote: On 07/08/2015 02:22 AM, Rafael Schloming wrote: a value of zero is actually what signals that the receiver needs to take some action here, and arguably an initial value of zero is correct since it is signaling that the receiver needs to take action (in this case issue credit). My interpretation is that if 0 is sent as the initial value, the sender cannot legally send any transfers without first expanding the window by sending a flow with a non-zero value. Agreed. We don't currently do that, so we are typically violating the window (Messenger being an exception, on its initial send to a node at least). In the case the remote incoming window is also 0 (which it is in the case which prompted this discussion) we would also have to synchronously wait for a flow 'response' increasing it before we could send anything, since we also need to know the remote incoming window actually enabled us to send. See my reply to Gordon for more details, but I think we need to be careful in distinguishing between the rules the spec mandates for updating/maintaining the value locally, and the rules the spec mandates for when it is necessary to communicate the value to the peer. These definitely should not be conflated because you don't want to notify the peer whenever you update the value locally, this would be way to chatty. Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). That was my take on reading things, that the two are essentially separate mechanisms, and on top the two windows are separate as well. If the initial outgoing window was set to 0 because the reciever hasnt issued, would there be a need for the field to exist on Begin? It would seem like much of the time it wouldnt be used and a Flow would have to be used instead. In the case where the sender's implementation involves a fixed amount of buffer space and requires messages to be settled before it can send more, the receiver would not be able to know that without getting some signal. So to my mind that is the only case for which it would make sense to send an outgoing window of 0. (I'm not sure how useful this is in practice and I don't believe it applies to proton at present anyway). I think as it stands proton is violating the spec, and should be changed to send a non-zero outgoing window. That is what I did for the proposed changes on https://issues.apache.org/jira/browse/PROTON-936 / https://github.com/apache/qpid-proton/pull/42. Essentially setting it initially on begin to max int (unless otherwise configured) and leaving it there for any subsequent flows. We could later (when we arent right before a release) look to make it smarter if needed. See my other email for more details, but I agree this seems like a reasonable change for now. --Rafael
Re: AMQP 1.0 session outgoing-window usage / meaning
On Wed, Jul 8, 2015 at 2:03 AM, Gordon Sim g...@redhat.com wrote: On 07/08/2015 02:22 AM, Rafael Schloming wrote: a value of zero is actually what signals that the receiver needs to take some action here, and arguably an initial value of zero is correct since it is signaling that the receiver needs to take action (in this case issue credit). My interpretation is that if 0 is sent as the initial value, the sender cannot legally send any transfers without first expanding the window by sending a flow with a non-zero value. I think what is confusing about the spec language is that it is defining the meaning of the value in terms of the sending endpoint's state and then not saying anything about when the sending endpoint is obligated to communicate the value to the receiver. This is because (at least as originally conceived) it is never directly obligated to communicate the value to the receiver, i.e. it could just choose to grow its array indefinitely. By contrast the incoming-window is defined in terms of the receiver's endpoint state and it is (relatively) clearly communicated that the receiver is obligated to communicate a non-zero incoming-window if it wishes to receive transfer frames. So while I agree that the language as defined requires the outgoing window to be increased from zero, I don't think this implies (at least without a more careful reading) that the sender needs to communicate this fact before sending transfers. It owns the value so it can effectively increase the outgoing-window to 1 momentarily, send the transfer, and then decrease it back to zero without needing to sandwich that transfer in flow frames in order to signal the changing window values. Put another way, the rule that says it is illegal for the sender to send transfers when the receiver's incoming window is zero creates an obligation for the receiver to communicate its window, but there is no corresponding rule for the receiver, e.g. nothing says it is illegal for the receiver to send credit when the sender's outgoing-window is zero, nor that the receiver must compute it's incoming-window based on the sender's outgoing-window, so there is no obligation similarly implied for the sender. Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). I agree that we shouldn't do this in our implementation, however I think it is a valid interpretation, which implies that making the incoming window dependent on your peers outgoing window (as service-bus is doing) is probably not a safe thing to do. In the case where the sender's implementation involves a fixed amount of buffer space and requires messages to be settled before it can send more, the receiver would not be able to know that without getting some signal. So to my mind that is the only case for which it would make sense to send an outgoing window of 0. (I'm not sure how useful this is in practice and I don't believe it applies to proton at present anyway). I think as it stands proton is violating the spec, and should be changed to send a non-zero outgoing window. Per above I don't believe it is violating the spec, but given that it has been misinterpreted at least once, I certainly agree that some behavior change is warranted. I don't think we should do the max-frame-size thing though as this encourages the notion that the incoming-window is somehow dependent on the outgoing-window which as I said above is I think unsafe. I guess setting it to a large constant for now is fairly reasonable, but I do think we should encourage service-bus to make its implementation more robust in this regard. --Rafael
Re: AMQP 1.0 session outgoing-window usage / meaning
On 8 July 2015 at 15:49, Rafael Schloming r...@alum.mit.edu wrote: On Wed, Jul 8, 2015 at 2:03 AM, Gordon Sim g...@redhat.com wrote: On 07/08/2015 02:22 AM, Rafael Schloming wrote: a value of zero is actually what signals that the receiver needs to take some action here, and arguably an initial value of zero is correct since it is signaling that the receiver needs to take action (in this case issue credit). My interpretation is that if 0 is sent as the initial value, the sender cannot legally send any transfers without first expanding the window by sending a flow with a non-zero value. I think what is confusing about the spec language is that it is defining the meaning of the value in terms of the sending endpoint's state and then not saying anything about when the sending endpoint is obligated to communicate the value to the receiver. This is because (at least as originally conceived) it is never directly obligated to communicate the value to the receiver, i.e. it could just choose to grow its array indefinitely. By contrast the incoming-window is defined in terms of the receiver's endpoint state and it is (relatively) clearly communicated that the receiver is obligated to communicate a non-zero incoming-window if it wishes to receive transfer frames. So while I agree that the language as defined requires the outgoing window to be increased from zero, I don't think this implies (at least without a more careful reading) that the sender needs to communicate this fact before sending transfers. It owns the value so it can effectively increase the outgoing-window to 1 momentarily, send the transfer, and then decrease it back to zero without needing to sandwich that transfer in flow frames in order to signal the changing window values. Put another way, the rule that says it is illegal for the sender to send transfers when the receiver's incoming window is zero creates an obligation for the receiver to communicate its window, but there is no corresponding rule for the receiver, e.g. nothing says it is illegal for the receiver to send credit when the sender's outgoing-window is zero, nor that the receiver must compute it's incoming-window based on the sender's outgoing-window, so there is no obligation similarly implied for the sender. The wording of This identifies a current maximum outgoing transfer-id that can be computed by subtracting one from the sum of outgoing-window and next-outgoing-id. when describing it, coupled with the requirement for the peer session to track it and decrement the remote-outgoing-window when receiving transfers, does suggest to me that things above the advertised point should not be sent (without first asynchronously communicating an increaseat least) since it would define a 'maximum' below the next-outgoing-id. On the other hand, you are right that it doesnt explicitly define when we need to update the peer, and there is a specific error condition symbol for when the peer exceeds the incoming window but there is no equivalent condition for doing the same with the outgoing window. I think the wording is wooly enough we could be here for a while figuring it out and still end up being unsure what it actually says though :) Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). I agree that we shouldn't do this in our implementation, however I think it is a valid interpretation, which implies that making the incoming window dependent on your peers outgoing window (as service-bus is doing) is probably not a safe thing to do. In the case where the sender's implementation involves a fixed amount of buffer space and requires messages to be settled before it can send more, the receiver would not be able to know that without getting some signal. So to my mind that is the only case for which it would make sense to send an outgoing window of 0. (I'm not sure how useful this is in practice and I don't believe it applies to proton at present anyway). I think as it stands proton is violating the spec, and should be changed to send a non-zero outgoing window. Per above I don't believe it is violating the spec, but given that it has been misinterpreted at least once, I certainly agree that some behavior change is warranted. I think Gordon is meaning in regard to the fact that we send a transfer when our outgoing window is [initially] 0 without first increasing it to e.g.1, due to the use of the phrasing I mentioned earlier that defines a maximum id below our [initial] next outgoing id. If so, I tend to share his view on that. Ultimately I think we just set a big window and essentially forgot it exists for now, we aren't using it at all within proton currently after all. We might as well stop calculating the remote value given we never actually
Re: AMQP 1.0 session outgoing-window usage / meaning
On 07/08/2015 03:49 PM, Rafael Schloming wrote: I think what is confusing about the spec language is that it is defining the meaning of the value in terms of the sending endpoint's state and then not saying anything about when the sending endpoint is obligated to communicate the value to the receiver. This is because (at least as originally conceived) it is never directly obligated to communicate the value to the receiver, i.e. it could just choose to grow its array indefinitely. By contrast the incoming-window is defined in terms of the receiver's endpoint state and it is (relatively) clearly communicated that the receiver is obligated to communicate a non-zero incoming-window if it wishes to receive transfer frames. So while I agree that the language as defined requires the outgoing window to be increased from zero, I don't think this implies (at least without a more careful reading) that the sender needs to communicate this fact before sending transfers. It owns the value so it can effectively increase the outgoing-window to 1 momentarily, send the transfer, and then decrease it back to zero without needing to sandwich that transfer in flow frames in order to signal the changing window values. I think that is a bit of a stretch given the language the spec does use. I agree it doesn't explicitly state the rule in those terms, however it does say: The remote-outgoing-window reflects the maximum number of incoming transfers that MAY arrive without exceeding the remote endpoint’s outgoing-window. This value MUST be decremented after every incoming transfer frame is received, and recomputed when informed of the remote session endpoint state. I.e. the language suggests that a compliant receiver must do some work to keep track of the value and states that this value is tied to the maximum number of transfers that may arrive. Put another way, the rule that says it is illegal for the sender to send transfers when the receiver's incoming window is zero creates an obligation for the receiver to communicate its window, but there is no corresponding rule for the receiver, e.g. nothing says it is illegal for the receiver to send credit when the sender's outgoing-window is zero, nor that the receiver must compute it's incoming-window based on the sender's outgoing-window, I certainly agree that nothing ties the receivers incoming window to the senders outgoing window in any fixed way. so there is no obligation similarly implied for the sender. I disagree here. I think the fact that begin and flow have the outgoing-window field defined as mandatory, and the fact that there is specific (though bewildering) language that appears to mandate the tracking of the senders outgoing window by the receiver, *implies* (though does not explicitly spell out) that the sender is supposed to keep the receiver informed of its outgoing window. Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). I agree that we shouldn't do this in our implementation, however I think it is a valid interpretation, The flow frame describes the outgoing window field as defining: the maximum number of outgoing transfer frames that the endpoint could potentially currently send, if it was not constrained by restrictions imposed by its peer’s incoming-window. To me that explicitly states that the value of the senders outgoing window is not tied to the value of the receivers incoming window, i.e. that a value of 0 for the senders outgoing window cannot reasonably be interpreted as an expectation for the receiver to open its incoming window. I don't see a single sentence that would tie the session outgoing window to link level credit, which is an entirely distinct mechanism. The one thing the spec does say of the remote-outgoing-window is that settling outstanding transfers can cause the window to grow. Settling deliveries frees the sender from needing to track them which clearly *might* enable a constrained sender to then send some more. Likewise a received outcome might allow the sender to make room for further transfers (though this isn't stated anywhere I can see it would seem a reasonable interpretation). which implies that making the incoming window dependent on your peers outgoing window (as service-bus is doing) is probably not a safe thing to do. Tying the incoming window to the peers outgoing window can only make sense if the sender is expected to keep the receiver up to date with regards to the value of the outgoing window. If that is not the case, then I can't see that the receiver can deduce anything from the outgoing window except that possibly a value of 0 may be a hint that the sender is held up pending some undefined action on the receiver part. Even if the sender
Re: AMQP 1.0 session outgoing-window usage / meaning
On 8 July 2015 at 16:03, Rafael Schloming r...@alum.mit.edu wrote: On Wed, Jul 8, 2015 at 2:38 AM, Robbie Gemmell robbie.gemm...@gmail.com wrote: On 8 July 2015 at 10:03, Gordon Sim g...@redhat.com wrote: On 07/08/2015 02:22 AM, Rafael Schloming wrote: a value of zero is actually what signals that the receiver needs to take some action here, and arguably an initial value of zero is correct since it is signaling that the receiver needs to take action (in this case issue credit). My interpretation is that if 0 is sent as the initial value, the sender cannot legally send any transfers without first expanding the window by sending a flow with a non-zero value. Agreed. We don't currently do that, so we are typically violating the window (Messenger being an exception, on its initial send to a node at least). In the case the remote incoming window is also 0 (which it is in the case which prompted this discussion) we would also have to synchronously wait for a flow 'response' increasing it before we could send anything, since we also need to know the remote incoming window actually enabled us to send. See my reply to Gordon for more details, but I think we need to be careful in distinguishing between the rules the spec mandates for updating/maintaining the value locally, and the rules the spec mandates for when it is necessary to communicate the value to the peer. These definitely should not be conflated because you don't want to notify the peer whenever you update the value locally, this would be way to chatty. I agree, you wouldnt waat to do that. To reiterate, I was only saying you would have to do that when both local outgoing and remote incoming were 0, given you would need to hear back about the new remote incoming (to ensure it not being 0) before you could actually send. You could of course asyncronously increase your outgoing window normally if you needed to, it just woudlnt do any good in that special case, which is the scenario we found ourselves to prompt this discussion. Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). That was my take on reading things, that the two are essentially separate mechanisms, and on top the two windows are separate as well. If the initial outgoing window was set to 0 because the reciever hasnt issued, would there be a need for the field to exist on Begin? It would seem like much of the time it wouldnt be used and a Flow would have to be used instead. In the case where the sender's implementation involves a fixed amount of buffer space and requires messages to be settled before it can send more, the receiver would not be able to know that without getting some signal. So to my mind that is the only case for which it would make sense to send an outgoing window of 0. (I'm not sure how useful this is in practice and I don't believe it applies to proton at present anyway). I think as it stands proton is violating the spec, and should be changed to send a non-zero outgoing window. That is what I did for the proposed changes on https://issues.apache.org/jira/browse/PROTON-936 / https://github.com/apache/qpid-proton/pull/42. Essentially setting it initially on begin to max int (unless otherwise configured) and leaving it there for any subsequent flows. We could later (when we arent right before a release) look to make it smarter if needed. See my other email for more details, but I agree this seems like a reasonable change for now. --Rafael
Re: AMQP 1.0 session outgoing-window usage / meaning
I think one safe thing to do now is to ignore this field. I will update service bus to not setting incoming window based on the received outgoing window field. This is the only time we look at this value. On Wed, Jul 8, 2015 at 10:58 AM, Gordon Sim g...@redhat.com wrote: On 07/08/2015 06:23 PM, Rafael Schloming wrote: On Wed, Jul 8, 2015 at 8:58 AM, Gordon Sim g...@redhat.com wrote: Under your interpretation - i.e. where the sender can change the value of the outgoing window without ever having to inform the receiver - the outgoing window seems to have very little value except as a hint that the sender may be constrained by capacity limits on the number of deliveries/transfers it is tracking. To send an outgoing window of 0 when there have not yet even been any transfers seems to undermine even that limited value. I think the way to look at it under that interpretation is not that the sender is not notifying the peer, simply that it is notifying the peer asynchronously, and that the transfer is implicitly notifying the peer (just like it implicitly advances the next-outgoing-id). I'm not sure I understand. Asynchronous with respect to what? Are you saying that a transfer implicitly expands the remote-outgoing-window (but only if it was 0 before the transfer)? One of the few concrete rules specified concerning the outgoing window is that the remote-outgoing-window MUST be decremented after every incoming transfer frame is received, and recomputed when informed of the remote session endpoint state. If the last information we received was that the outgoing window was 0, what should we do on then receiving a transfer?
Re: AMQP 1.0 session outgoing-window usage / meaning
On 8 July 2015 at 17:59, Rafael Schloming r...@alum.mit.edu wrote: On Wed, Jul 8, 2015 at 8:29 AM, Robbie Gemmell robbie.gemm...@gmail.com wrote: The wording of This identifies a current maximum outgoing transfer-id that can be computed by subtracting one from the sum of outgoing-window and next-outgoing-id. when describing it, coupled with the requirement for the peer session to track it and decrement the remote-outgoing-window when receiving transfers, does suggest to me that things above the advertised point should not be sent (without first asynchronously communicating an increaseat least) since it would define a 'maximum' below the next-outgoing-id. On the other hand, you are right that it doesnt explicitly define when we need to update the peer, and there is a specific error condition symbol for when the peer exceeds the incoming window but there is no equivalent condition for doing the same with the outgoing window. I think the wording is wooly enough we could be here for a while figuring it out and still end up being unsure what it actually says though :) Fair enough, and I think the conservative thing to do is certainly to asynchronously notify your peer when the window changes, however as in my other reply I don't see how doing that actually eliminates the possibility of deadlock with service-bus. Asynchronously notifying your peer when you change the window is a very different requirement than being obliged to notify your peer that the window is nonzero in order to receive credit. For the ServiceBus case I was saying we would need to do it synchronously, which we obviously wouldnt want to for various reasons. That would also presumably require us (and SB) to support the echo flag, which I think I recall raising a JIRA about some time ago. I'm much more inclined to set the window high and forget it exists :) I think Gordon is meaning in regard to the fact that we send a transfer when our outgoing window is [initially] 0 without first increasing it to e.g.1, due to the use of the phrasing I mentioned earlier that defines a maximum id below our [initial] next outgoing id. If so, I tend to share his view on that. Gotcha, I certainly agree that the robust thing to do would be to issue the flow frame in such cases. Ultimately I think we just set a big window and essentially forgot it exists for now, we aren't using it at all within proton currently after all. We might as well stop calculating the remote value given we never actually read it. I don't think we should do the max-frame-size thing though as this encourages the notion that the incoming-window is somehow dependent on the outgoing-window which as I said above is I think unsafe. I guess setting it to a large constant for now is fairly reasonable, but I do think we should encourage service-bus to make its implementation more robust in this regard. --Rafael Can I take that as a +1 on the approach I proposed changes for on PROTON-936 / https://github.com/apache/qpid-proton/pull/42 ? Yes --Rafael Great, I'll push that change in tomorrow morning.
Re: AMQP 1.0 session outgoing-window usage / meaning
On Wed, Jul 8, 2015 at 8:58 AM, Gordon Sim g...@redhat.com wrote: On 07/08/2015 03:49 PM, Rafael Schloming wrote: I think what is confusing about the spec language is that it is defining the meaning of the value in terms of the sending endpoint's state and then not saying anything about when the sending endpoint is obligated to communicate the value to the receiver. This is because (at least as originally conceived) it is never directly obligated to communicate the value to the receiver, i.e. it could just choose to grow its array indefinitely. By contrast the incoming-window is defined in terms of the receiver's endpoint state and it is (relatively) clearly communicated that the receiver is obligated to communicate a non-zero incoming-window if it wishes to receive transfer frames. So while I agree that the language as defined requires the outgoing window to be increased from zero, I don't think this implies (at least without a more careful reading) that the sender needs to communicate this fact before sending transfers. It owns the value so it can effectively increase the outgoing-window to 1 momentarily, send the transfer, and then decrease it back to zero without needing to sandwich that transfer in flow frames in order to signal the changing window values. I think that is a bit of a stretch given the language the spec does use. I agree it doesn't explicitly state the rule in those terms, however it does say: The remote-outgoing-window reflects the maximum number of incoming transfers that MAY arrive without exceeding the remote endpoint’s outgoing-window. This value MUST be decremented after every incoming transfer frame is received, and recomputed when informed of the remote session endpoint state. I.e. the language suggests that a compliant receiver must do some work to keep track of the value and states that this value is tied to the maximum number of transfers that may arrive. Put another way, the rule that says it is illegal for the sender to send transfers when the receiver's incoming window is zero creates an obligation for the receiver to communicate its window, but there is no corresponding rule for the receiver, e.g. nothing says it is illegal for the receiver to send credit when the sender's outgoing-window is zero, nor that the receiver must compute it's incoming-window based on the sender's outgoing-window, I certainly agree that nothing ties the receivers incoming window to the senders outgoing window in any fixed way. so there is no obligation similarly implied for the sender. I disagree here. I think the fact that begin and flow have the outgoing-window field defined as mandatory, and the fact that there is specific (though bewildering) language that appears to mandate the tracking of the senders outgoing window by the receiver, *implies* (though does not explicitly spell out) that the sender is supposed to keep the receiver informed of its outgoing window. Further I think the sender should not take the lack of credit as grounds to set a window of 0. The receiver knows it has not issued credit. (At the link level, the sender can also indicate that it has messages awaiting link credit). I agree that we shouldn't do this in our implementation, however I think it is a valid interpretation, The flow frame describes the outgoing window field as defining: the maximum number of outgoing transfer frames that the endpoint could potentially currently send, if it was not constrained by restrictions imposed by its peer’s incoming-window. To me that explicitly states that the value of the senders outgoing window is not tied to the value of the receivers incoming window, i.e. that a value of 0 for the senders outgoing window cannot reasonably be interpreted as an expectation for the receiver to open its incoming window. I don't see a single sentence that would tie the session outgoing window to link level credit, which is an entirely distinct mechanism. The one thing the spec does say of the remote-outgoing-window is that settling outstanding transfers can cause the window to grow. Settling deliveries frees the sender from needing to track them which clearly *might* enable a constrained sender to then send some more. Likewise a received outcome might allow the sender to make room for further transfers (though this isn't stated anywhere I can see it would seem a reasonable interpretation). which implies that making the incoming window dependent on your peers outgoing window (as service-bus is doing) is probably not a safe thing to do. Tying the incoming window to the peers outgoing window can only make sense if the sender is expected to keep the receiver up to date with regards to the value of the outgoing window. If that is not the case, then I can't see that the receiver can deduce anything from the outgoing window except that possibly a value of
Re: AMQP 1.0 session outgoing-window usage / meaning
On Wed, Jul 8, 2015 at 8:29 AM, Robbie Gemmell robbie.gemm...@gmail.com wrote: The wording of This identifies a current maximum outgoing transfer-id that can be computed by subtracting one from the sum of outgoing-window and next-outgoing-id. when describing it, coupled with the requirement for the peer session to track it and decrement the remote-outgoing-window when receiving transfers, does suggest to me that things above the advertised point should not be sent (without first asynchronously communicating an increaseat least) since it would define a 'maximum' below the next-outgoing-id. On the other hand, you are right that it doesnt explicitly define when we need to update the peer, and there is a specific error condition symbol for when the peer exceeds the incoming window but there is no equivalent condition for doing the same with the outgoing window. I think the wording is wooly enough we could be here for a while figuring it out and still end up being unsure what it actually says though :) Fair enough, and I think the conservative thing to do is certainly to asynchronously notify your peer when the window changes, however as in my other reply I don't see how doing that actually eliminates the possibility of deadlock with service-bus. Asynchronously notifying your peer when you change the window is a very different requirement than being obliged to notify your peer that the window is nonzero in order to receive credit. I think Gordon is meaning in regard to the fact that we send a transfer when our outgoing window is [initially] 0 without first increasing it to e.g.1, due to the use of the phrasing I mentioned earlier that defines a maximum id below our [initial] next outgoing id. If so, I tend to share his view on that. Gotcha, I certainly agree that the robust thing to do would be to issue the flow frame in such cases. Ultimately I think we just set a big window and essentially forgot it exists for now, we aren't using it at all within proton currently after all. We might as well stop calculating the remote value given we never actually read it. I don't think we should do the max-frame-size thing though as this encourages the notion that the incoming-window is somehow dependent on the outgoing-window which as I said above is I think unsafe. I guess setting it to a large constant for now is fairly reasonable, but I do think we should encourage service-bus to make its implementation more robust in this regard. --Rafael Can I take that as a +1 on the approach I proposed changes for on PROTON-936 / https://github.com/apache/qpid-proton/pull/42 ? Yes --Rafael
Re: AMQP 1.0 session outgoing-window usage / meaning
On Wed, Jul 8, 2015 at 8:28 AM, iPC daron...@gmail.com wrote: // quotes from the spec --- 2.5.6 Session Flow Control The Session Endpoint assigns each outgoing transfer frame an implicit transfer-id from a session scoped sequence. outgoing-window: The outgoing-window defines the maximum number of outgoing transfer frames that the endpoint can currently send. This identifies a current maximum outgoing transfer-id that can be computed by subtracting one from the sum of outgoing-window and next-outgoing-id. // end quotes --- Outgoing-window is a mandatory field on flow. The sender's outgoing window is communicated to the peer. If the sender sends a 0 outgoing window but subsequently sends a transfer frame, the implicit outgoing transfer-id would exceed the limit implied by the flow frame. I agree that the sender can maintain a local outgoing window to satisfy any local implementation specific logic, but the value communicated to the peer must allow further transfer frames to be sent. I don't think it makes senses to use 0 outgoing window to request for credit (assuming it is the link credit mentioned in early mails). Using a 0 link credit would make more sense. That's certainly a fair interpretation, but even under that interpretation a potential deadlock with service-bus may still exist. Consider an implementation that sets its initial outgoing window to zero, waits for credit, and then upon receiving credit expands its window, sends a flow frame with the updated window, and then sends a transfer. I expect this implementation would deadlock with service-bus, but I don't think it is violating the spec under any of the interpretations proposed in this thread. --Rafael
Re: AMQP 1.0 session outgoing-window usage / meaning
On 07/08/2015 06:23 PM, Rafael Schloming wrote: On Wed, Jul 8, 2015 at 8:58 AM, Gordon Sim g...@redhat.com wrote: Under your interpretation - i.e. where the sender can change the value of the outgoing window without ever having to inform the receiver - the outgoing window seems to have very little value except as a hint that the sender may be constrained by capacity limits on the number of deliveries/transfers it is tracking. To send an outgoing window of 0 when there have not yet even been any transfers seems to undermine even that limited value. I think the way to look at it under that interpretation is not that the sender is not notifying the peer, simply that it is notifying the peer asynchronously, and that the transfer is implicitly notifying the peer (just like it implicitly advances the next-outgoing-id). I'm not sure I understand. Asynchronous with respect to what? Are you saying that a transfer implicitly expands the remote-outgoing-window (but only if it was 0 before the transfer)? One of the few concrete rules specified concerning the outgoing window is that the remote-outgoing-window MUST be decremented after every incoming transfer frame is received, and recomputed when informed of the remote session endpoint state. If the last information we received was that the outgoing window was 0, what should we do on then receiving a transfer?
Re: AMQP 1.0 session outgoing-window usage / meaning
On 8 July 2015 at 20:13, Rob Godfrey rob.j.godf...@gmail.com wrote: As far as I can recall/reconstruct the only utility given by the outgoing window was so that the sender (of transfer frames) can indicate to the receiver (of transfer frames) that it will require notification of which frames have been seen by the receiver within window size frames, otherwise the sender will potentially block until it has received such information. As such the receiver should always be able to safely set its incoming window to be = the sender's outgoing window, since setting it to be larger is pointless. A sender which finds itself having previously set the outgoing window size to zero, and finding its peer's incoming window size is also zero must, I think, send a flow indicating that its window size is now non-zero. Having said that I'm not sure I ever saw much utility in this value and as such I would agree that implementations should probably just set the default outgoing window size to a large number, and ignore their peer's outgoing window when setting their own incoming window. About the only thing that we probably should be doing is ensuring that if the implied outgoing window of the peer is zero, but might open up if a flow informing the peer of the updated state, then such a flow should be sent. (Not sure I would set to MAX_INT in case anyone is trying to create an array based on the provided size upon receipt... but that's just me). -- Rob I did wonder if setting it that large was too large (though its only half the allowed value)...but decided its a bit difficult to pick a 'large enough to forget it exists, but not too large' number, and just ensured it is configurable in case needed. On 8 July 2015 at 20:46, Xin Chen daron...@gmail.com wrote: I think one safe thing to do now is to ignore this field. I will update service bus to not setting incoming window based on the received outgoing window field. This is the only time we look at this value. On Wed, Jul 8, 2015 at 10:58 AM, Gordon Sim g...@redhat.com wrote: On 07/08/2015 06:23 PM, Rafael Schloming wrote: On Wed, Jul 8, 2015 at 8:58 AM, Gordon Sim g...@redhat.com wrote: Under your interpretation - i.e. where the sender can change the value of the outgoing window without ever having to inform the receiver - the outgoing window seems to have very little value except as a hint that the sender may be constrained by capacity limits on the number of deliveries/transfers it is tracking. To send an outgoing window of 0 when there have not yet even been any transfers seems to undermine even that limited value. I think the way to look at it under that interpretation is not that the sender is not notifying the peer, simply that it is notifying the peer asynchronously, and that the transfer is implicitly notifying the peer (just like it implicitly advances the next-outgoing-id). I'm not sure I understand. Asynchronous with respect to what? Are you saying that a transfer implicitly expands the remote-outgoing-window (but only if it was 0 before the transfer)? One of the few concrete rules specified concerning the outgoing window is that the remote-outgoing-window MUST be decremented after every incoming transfer frame is received, and recomputed when informed of the remote session endpoint state. If the last information we received was that the outgoing window was 0, what should we do on then receiving a transfer?
Re: AMQP 1.0 session outgoing-window usage / meaning
As far as I can recall/reconstruct the only utility given by the outgoing window was so that the sender (of transfer frames) can indicate to the receiver (of transfer frames) that it will require notification of which frames have been seen by the receiver within window size frames, otherwise the sender will potentially block until it has received such information. As such the receiver should always be able to safely set its incoming window to be = the sender's outgoing window, since setting it to be larger is pointless. A sender which finds itself having previously set the outgoing window size to zero, and finding its peer's incoming window size is also zero must, I think, send a flow indicating that its window size is now non-zero. Having said that I'm not sure I ever saw much utility in this value and as such I would agree that implementations should probably just set the default outgoing window size to a large number, and ignore their peer's outgoing window when setting their own incoming window. About the only thing that we probably should be doing is ensuring that if the implied outgoing window of the peer is zero, but might open up if a flow informing the peer of the updated state, then such a flow should be sent. (Not sure I would set to MAX_INT in case anyone is trying to create an array based on the provided size upon receipt... but that's just me). -- Rob On 8 July 2015 at 20:46, Xin Chen daron...@gmail.com wrote: I think one safe thing to do now is to ignore this field. I will update service bus to not setting incoming window based on the received outgoing window field. This is the only time we look at this value. On Wed, Jul 8, 2015 at 10:58 AM, Gordon Sim g...@redhat.com wrote: On 07/08/2015 06:23 PM, Rafael Schloming wrote: On Wed, Jul 8, 2015 at 8:58 AM, Gordon Sim g...@redhat.com wrote: Under your interpretation - i.e. where the sender can change the value of the outgoing window without ever having to inform the receiver - the outgoing window seems to have very little value except as a hint that the sender may be constrained by capacity limits on the number of deliveries/transfers it is tracking. To send an outgoing window of 0 when there have not yet even been any transfers seems to undermine even that limited value. I think the way to look at it under that interpretation is not that the sender is not notifying the peer, simply that it is notifying the peer asynchronously, and that the transfer is implicitly notifying the peer (just like it implicitly advances the next-outgoing-id). I'm not sure I understand. Asynchronous with respect to what? Are you saying that a transfer implicitly expands the remote-outgoing-window (but only if it was 0 before the transfer)? One of the few concrete rules specified concerning the outgoing window is that the remote-outgoing-window MUST be decremented after every incoming transfer frame is received, and recomputed when informed of the remote session endpoint state. If the last information we received was that the outgoing window was 0, what should we do on then receiving a transfer?
Re: AMQP 1.0 session outgoing-window usage / meaning
IIRC, the definition of the outgoing-window was largely motivated by the need to express to receivers certain conditions under which they may be required to settle deliveries in order to receive more. For example if an implementation uses a fixed sized array to store deliveries, and this array is keyed by the offset of the delivery-id from the smallest unsettled delivery, then although the sender may have sufficient credit to send more transfers, it may not actually be capable of doing this because the next delivery-id would land outside the range of deliveries that are currently represented within its fixed size array. This could happen for example if the receiver issues N credits (where N is the size of the sender's fixed array) and settles deliveries 2 through N. The sender is then stuck with an unsettled delivery in the first slot of its fixed sized array and cannot send another delivery until that first delivery is settled. Given this, it's certainly true an outgoing-window of 0 is kind of strange and useless. It's probably also true that it is never super useful for the incoming window of the receiver to be larger than the outgoing window of the sender (or vice versa) since one can't ever exceed the other, so I'd say your largest-possible-int default and max-frame-like treatment are fairly appropriate. --Rafael On Fri, Jul 3, 2015 at 8:57 AM, Robbie Gemmell robbie.gemm...@gmail.com wrote: Rob, Rafi, as authors of the spec and the related code in proton, do you have any thoughts to add here? Barring any discussion otherwise I will be looking to change proton to at least optionally allow controlling the outgoing window along the lines I mentioned near the end of my original mail. Robbie On 2 July 2015 at 00:15, Robbie Gemmell robbie.gemm...@gmail.com wrote: Thanks James. Some expansion which may be useful to add. When comparing the older JMS client, proton-c via the Messenger API, and the new JMS client using proton-j, its important to note that they aren't all doing the same thing even where their underlying implementations do seem to share the same behaviour in the cases of proton-c and proton-j. The older JMS client initializes its outgoing window to a fixed number in the session Begin frame and then doesnt seem to ever change it for subsequent Flow frames, and simply manages whether its session can later send transfer frames based on the current value of the remote incoming window. Proton-J and Proton-C similarly only base their session level decision to send transfers on the remote incoming window and not their own outgoing window (which as noted below means they violate their advertised outgoing window, which is often going to be 0). Proton-C and Proton-J both currently look to set the outgoing window at any given time to a calculated value based on either the number of buffered messages or the buffered bytes divided by frame size. If there are no buffered messages at the point the Begin and Flow frames are generated, then the outgoing-window will be set to 0. This appears to function the same for both proton-c and proton-j. A key point though is that I think much of the historic usage of proton-c against Service Bus has been via the Messenger API, which works somewhat differently than many others in that it looks to create a session and a sender and sends the messages in one pipelined sequence of transport output, which means that by the point the Begin frame actually gets generated there are indeed buffered messages to send which means the outgoing-window is initialised to a value greater than zero. Other APIs which create the session as a distinct step thus wont ever have buffered messages when the Begin frame gets created and so the outgoing-window is initialised to 0, which is the behaviour observed with the new JMS client using proton-j and also what I saw when trying proton-c via the Qpid Messaging C++ client (against qpidd). Robbie On 1 July 2015 at 20:54, James Birdsall jb...@microsoft.com wrote: FYI, I have forwarded this and important bits of the preceding discussion to our AMQP stack dev within the ServiceBus team. Both the Qpid JMS AMQP 1.0 legacy client and Proton-C have been working fine with Azure SB for years now. Proton-J, however, is not something we have explored previously, and obviously there is something different about its behavior compared to the other clients. The Qpid JMS client is our recommended JMS client for interop with ServiceBus, and we would like to keep up with the times and not have to direct customers to the legacy client, so we are very interested in figuring out the correct resolution to this issue. -Original Message- From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] Sent: Wednesday, July 1, 2015 7:48 AM To: us...@qpid.apache.org; proton@qpid.apache.org Subject: AMQP 1.0 session outgoing-window usage / meaning Hi
Re: AMQP 1.0 session outgoing-window usage / meaning
On 07/07/2015 07:22 AM, Rafael Schloming wrote: IIRC, the definition of the outgoing-window was largely motivated by the need to express to receivers certain conditions under which they may be required to settle deliveries in order to receive more. For example if an implementation uses a fixed sized array to store deliveries, and this array is keyed by the offset of the delivery-id from the smallest unsettled delivery, then although the sender may have sufficient credit to send more transfers, it may not actually be capable of doing this because the next delivery-id would land outside the range of deliveries that are currently represented within its fixed size array. The outgoing-window is measured in transfers, right? So in this case each slot in the array would be a *transfer* with a single delivery possibly spanning multiple slots. This could happen for example if the receiver issues N credits (where N is the size of the sender's fixed array) and settles deliveries 2 through N. The sender is then stuck with an unsettled delivery in the first slot of its fixed sized array and cannot send another delivery until that first delivery is settled. Given this, it's certainly true an outgoing-window of 0 is kind of strange and useless. Isn't that exactly the mechanism by which a sender, such as the one in your description above, would indicate its inability to send further transfers? It's probably also true that it is never super useful for the incoming window of the receiver to be larger than the outgoing window of the sender (or vice versa) since one can't ever exceed the other, so I'd say your largest-possible-int default and max-frame-like treatment are fairly appropriate.
Re: AMQP 1.0 session outgoing-window usage / meaning
On Tue, Jul 7, 2015 at 4:29 AM, Gordon Sim g...@redhat.com wrote: On 07/07/2015 07:22 AM, Rafael Schloming wrote: IIRC, the definition of the outgoing-window was largely motivated by the need to express to receivers certain conditions under which they may be required to settle deliveries in order to receive more. For example if an implementation uses a fixed sized array to store deliveries, and this array is keyed by the offset of the delivery-id from the smallest unsettled delivery, then although the sender may have sufficient credit to send more transfers, it may not actually be capable of doing this because the next delivery-id would land outside the range of deliveries that are currently represented within its fixed size array. The outgoing-window is measured in transfers, right? So in this case each slot in the array would be a *transfer* with a single delivery possibly spanning multiple slots. Yes, good point. This could happen for example if the receiver issues N credits (where N is the size of the sender's fixed array) and settles deliveries 2 through N. The sender is then stuck with an unsettled delivery in the first slot of its fixed sized array and cannot send another delivery until that first delivery is settled. Given this, it's certainly true an outgoing-window of 0 is kind of strange and useless. Isn't that exactly the mechanism by which a sender, such as the one in your description above, would indicate its inability to send further transfers? Gah, sorry, jet lag... you are right. I was thinking of the window measured from the oldest unsettled transfers, however it is actually measured from the next-outgoing-id, and so as you say a value of zero is actually what signals that the receiver needs to take some action here, and arguably an initial value of zero is correct since it is signaling that the receiver needs to take action (in this case issue credit). --Rafael
Re: AMQP 1.0 session outgoing-window usage / meaning
bits of the preceding discussion to our AMQP stack dev within the ServiceBus team. Both the Qpid JMS AMQP 1.0 legacy client and Proton-C have been working fine with Azure SB for years now. Proton-J, however, is not something we have explored previously, and obviously there is something different about its behavior compared to the other clients. The Qpid JMS client is our recommended JMS client for interop with ServiceBus, and we would like to keep up with the times and not have to direct customers to the legacy client, so we are very interested in figuring out the correct resolution to this issue. -Original Message- From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] Sent: Wednesday, July 1, 2015 7:48 AM To: us...@qpid.apache.org; proton@qpid.apache.org Subject: AMQP 1.0 session outgoing-window usage / meaning Hi all, Short intro: The way we use the outgoing-window feels wrong, and seems to violate at least one bit of the related [and unclear overall] description in the spec. The way we use it means we currently can't send messages to ServiceBus in many cases (likely anything-but-messenger). Full version: A user reported being unable to send messages to Service Bus (Azure or Windows variants) using the new JMS client. Investigating that lead me to the handling of session outgoing-window in proton (and incoming-window in ServiceBus). I'd like to discuss how proton uses this, what the spec actually says on how it should be used since its not clear to me these are currently in alignment, and possibly changing how we utilise it in proton. In Proton the outgoing-window is set based on the amount of outstanding outgoing bytes for the session (from buffered sends) and the frame size, which leads to many situations where the advertised value is 0, particularly in a client that creates sessions entirely independently from sending messages (such as a JMS client, or the Qpid Messaging C++ client). We then send transfer frames if a link has credit, regardless of the session outgoing window, only withholding them if the remote-incoming-window hits 0. This causes problems against ServiceBus because it seems to base its advertised incoming window on the clients [initial] advertised outgoing window. If it is 0, then it means we can never send any messages. The intent/definition of the outgoing-window seems a bit unclear as a whole in the spec to me, to the point you could almost remove it without issue given the [remote-]incoming-window seems to govern overall behaviour. However, it seems wrong to me that we often set the outgoing-window to 0. Doing this means we violate the outgoing window (local and at peer) whenever we send a message if the remote-incoming-window allowed it. If we tried not to violate it, we would have to flow the new session window before every send, which again seems odd. It feels to me like it would be better to define initial values for the windows, perhaps allowing that to be configurable in the same manner as e.g max frame size is. Those that want to impose some limit then could, and those that don't can leave/set it to the max values to achieve that effect. Thoughts? Robbie - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: AMQP 1.0 session outgoing-window usage / meaning
Rob, Rafi, as authors of the spec and the related code in proton, do you have any thoughts to add here? Barring any discussion otherwise I will be looking to change proton to at least optionally allow controlling the outgoing window along the lines I mentioned near the end of my original mail. Robbie On 2 July 2015 at 00:15, Robbie Gemmell robbie.gemm...@gmail.com wrote: Thanks James. Some expansion which may be useful to add. When comparing the older JMS client, proton-c via the Messenger API, and the new JMS client using proton-j, its important to note that they aren't all doing the same thing even where their underlying implementations do seem to share the same behaviour in the cases of proton-c and proton-j. The older JMS client initializes its outgoing window to a fixed number in the session Begin frame and then doesnt seem to ever change it for subsequent Flow frames, and simply manages whether its session can later send transfer frames based on the current value of the remote incoming window. Proton-J and Proton-C similarly only base their session level decision to send transfers on the remote incoming window and not their own outgoing window (which as noted below means they violate their advertised outgoing window, which is often going to be 0). Proton-C and Proton-J both currently look to set the outgoing window at any given time to a calculated value based on either the number of buffered messages or the buffered bytes divided by frame size. If there are no buffered messages at the point the Begin and Flow frames are generated, then the outgoing-window will be set to 0. This appears to function the same for both proton-c and proton-j. A key point though is that I think much of the historic usage of proton-c against Service Bus has been via the Messenger API, which works somewhat differently than many others in that it looks to create a session and a sender and sends the messages in one pipelined sequence of transport output, which means that by the point the Begin frame actually gets generated there are indeed buffered messages to send which means the outgoing-window is initialised to a value greater than zero. Other APIs which create the session as a distinct step thus wont ever have buffered messages when the Begin frame gets created and so the outgoing-window is initialised to 0, which is the behaviour observed with the new JMS client using proton-j and also what I saw when trying proton-c via the Qpid Messaging C++ client (against qpidd). Robbie On 1 July 2015 at 20:54, James Birdsall jb...@microsoft.com wrote: FYI, I have forwarded this and important bits of the preceding discussion to our AMQP stack dev within the ServiceBus team. Both the Qpid JMS AMQP 1.0 legacy client and Proton-C have been working fine with Azure SB for years now. Proton-J, however, is not something we have explored previously, and obviously there is something different about its behavior compared to the other clients. The Qpid JMS client is our recommended JMS client for interop with ServiceBus, and we would like to keep up with the times and not have to direct customers to the legacy client, so we are very interested in figuring out the correct resolution to this issue. -Original Message- From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] Sent: Wednesday, July 1, 2015 7:48 AM To: us...@qpid.apache.org; proton@qpid.apache.org Subject: AMQP 1.0 session outgoing-window usage / meaning Hi all, Short intro: The way we use the outgoing-window feels wrong, and seems to violate at least one bit of the related [and unclear overall] description in the spec. The way we use it means we currently can't send messages to ServiceBus in many cases (likely anything-but-messenger). Full version: A user reported being unable to send messages to Service Bus (Azure or Windows variants) using the new JMS client. Investigating that lead me to the handling of session outgoing-window in proton (and incoming-window in ServiceBus). I'd like to discuss how proton uses this, what the spec actually says on how it should be used since its not clear to me these are currently in alignment, and possibly changing how we utilise it in proton. In Proton the outgoing-window is set based on the amount of outstanding outgoing bytes for the session (from buffered sends) and the frame size, which leads to many situations where the advertised value is 0, particularly in a client that creates sessions entirely independently from sending messages (such as a JMS client, or the Qpid Messaging C++ client). We then send transfer frames if a link has credit, regardless of the session outgoing window, only withholding them if the remote-incoming-window hits 0. This causes problems against ServiceBus because it seems to base its advertised incoming window on the clients [initial] advertised outgoing window. If it is 0, then it means we can
AMQP 1.0 session outgoing-window usage / meaning
Hi all, Short intro: The way we use the outgoing-window feels wrong, and seems to violate at least one bit of the related [and unclear overall] description in the spec. The way we use it means we currently can't send messages to ServiceBus in many cases (likely anything-but-messenger). Full version: A user reported being unable to send messages to Service Bus (Azure or Windows variants) using the new JMS client. Investigating that lead me to the handling of session outgoing-window in proton (and incoming-window in ServiceBus). I'd like to discuss how proton uses this, what the spec actually says on how it should be used since its not clear to me these are currently in alignment, and possibly changing how we utilise it in proton. In Proton the outgoing-window is set based on the amount of outstanding outgoing bytes for the session (from buffered sends) and the frame size, which leads to many situations where the advertised value is 0, particularly in a client that creates sessions entirely independently from sending messages (such as a JMS client, or the Qpid Messaging C++ client). We then send transfer frames if a link has credit, regardless of the session outgoing window, only withholding them if the remote-incoming-window hits 0. This causes problems against ServiceBus because it seems to base its advertised incoming window on the clients [initial] advertised outgoing window. If it is 0, then it means we can never send any messages. The intent/definition of the outgoing-window seems a bit unclear as a whole in the spec to me, to the point you could almost remove it without issue given the [remote-]incoming-window seems to govern overall behaviour. However, it seems wrong to me that we often set the outgoing-window to 0. Doing this means we violate the outgoing window (local and at peer) whenever we send a message if the remote-incoming-window allowed it. If we tried not to violate it, we would have to flow the new session window before every send, which again seems odd. It feels to me like it would be better to define initial values for the windows, perhaps allowing that to be configurable in the same manner as e.g max frame size is. Those that want to impose some limit then could, and those that don't can leave/set it to the max values to achieve that effect. Thoughts? Robbie
RE: AMQP 1.0 session outgoing-window usage / meaning
FYI, I have forwarded this and important bits of the preceding discussion to our AMQP stack dev within the ServiceBus team. Both the Qpid JMS AMQP 1.0 legacy client and Proton-C have been working fine with Azure SB for years now. Proton-J, however, is not something we have explored previously, and obviously there is something different about its behavior compared to the other clients. The Qpid JMS client is our recommended JMS client for interop with ServiceBus, and we would like to keep up with the times and not have to direct customers to the legacy client, so we are very interested in figuring out the correct resolution to this issue. -Original Message- From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] Sent: Wednesday, July 1, 2015 7:48 AM To: us...@qpid.apache.org; proton@qpid.apache.org Subject: AMQP 1.0 session outgoing-window usage / meaning Hi all, Short intro: The way we use the outgoing-window feels wrong, and seems to violate at least one bit of the related [and unclear overall] description in the spec. The way we use it means we currently can't send messages to ServiceBus in many cases (likely anything-but-messenger). Full version: A user reported being unable to send messages to Service Bus (Azure or Windows variants) using the new JMS client. Investigating that lead me to the handling of session outgoing-window in proton (and incoming-window in ServiceBus). I'd like to discuss how proton uses this, what the spec actually says on how it should be used since its not clear to me these are currently in alignment, and possibly changing how we utilise it in proton. In Proton the outgoing-window is set based on the amount of outstanding outgoing bytes for the session (from buffered sends) and the frame size, which leads to many situations where the advertised value is 0, particularly in a client that creates sessions entirely independently from sending messages (such as a JMS client, or the Qpid Messaging C++ client). We then send transfer frames if a link has credit, regardless of the session outgoing window, only withholding them if the remote-incoming-window hits 0. This causes problems against ServiceBus because it seems to base its advertised incoming window on the clients [initial] advertised outgoing window. If it is 0, then it means we can never send any messages. The intent/definition of the outgoing-window seems a bit unclear as a whole in the spec to me, to the point you could almost remove it without issue given the [remote-]incoming-window seems to govern overall behaviour. However, it seems wrong to me that we often set the outgoing-window to 0. Doing this means we violate the outgoing window (local and at peer) whenever we send a message if the remote-incoming-window allowed it. If we tried not to violate it, we would have to flow the new session window before every send, which again seems odd. It feels to me like it would be better to define initial values for the windows, perhaps allowing that to be configurable in the same manner as e.g max frame size is. Those that want to impose some limit then could, and those that don't can leave/set it to the max values to achieve that effect. Thoughts? Robbie
Re: AMQP 1.0 session outgoing-window usage / meaning
Thanks James. Some expansion which may be useful to add. When comparing the older JMS client, proton-c via the Messenger API, and the new JMS client using proton-j, its important to note that they aren't all doing the same thing even where their underlying implementations do seem to share the same behaviour in the cases of proton-c and proton-j. The older JMS client initializes its outgoing window to a fixed number in the session Begin frame and then doesnt seem to ever change it for subsequent Flow frames, and simply manages whether its session can later send transfer frames based on the current value of the remote incoming window. Proton-J and Proton-C similarly only base their session level decision to send transfers on the remote incoming window and not their own outgoing window (which as noted below means they violate their advertised outgoing window, which is often going to be 0). Proton-C and Proton-J both currently look to set the outgoing window at any given time to a calculated value based on either the number of buffered messages or the buffered bytes divided by frame size. If there are no buffered messages at the point the Begin and Flow frames are generated, then the outgoing-window will be set to 0. This appears to function the same for both proton-c and proton-j. A key point though is that I think much of the historic usage of proton-c against Service Bus has been via the Messenger API, which works somewhat differently than many others in that it looks to create a session and a sender and sends the messages in one pipelined sequence of transport output, which means that by the point the Begin frame actually gets generated there are indeed buffered messages to send which means the outgoing-window is initialised to a value greater than zero. Other APIs which create the session as a distinct step thus wont ever have buffered messages when the Begin frame gets created and so the outgoing-window is initialised to 0, which is the behaviour observed with the new JMS client using proton-j and also what I saw when trying proton-c via the Qpid Messaging C++ client (against qpidd). Robbie On 1 July 2015 at 20:54, James Birdsall jb...@microsoft.com wrote: FYI, I have forwarded this and important bits of the preceding discussion to our AMQP stack dev within the ServiceBus team. Both the Qpid JMS AMQP 1.0 legacy client and Proton-C have been working fine with Azure SB for years now. Proton-J, however, is not something we have explored previously, and obviously there is something different about its behavior compared to the other clients. The Qpid JMS client is our recommended JMS client for interop with ServiceBus, and we would like to keep up with the times and not have to direct customers to the legacy client, so we are very interested in figuring out the correct resolution to this issue. -Original Message- From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] Sent: Wednesday, July 1, 2015 7:48 AM To: us...@qpid.apache.org; proton@qpid.apache.org Subject: AMQP 1.0 session outgoing-window usage / meaning Hi all, Short intro: The way we use the outgoing-window feels wrong, and seems to violate at least one bit of the related [and unclear overall] description in the spec. The way we use it means we currently can't send messages to ServiceBus in many cases (likely anything-but-messenger). Full version: A user reported being unable to send messages to Service Bus (Azure or Windows variants) using the new JMS client. Investigating that lead me to the handling of session outgoing-window in proton (and incoming-window in ServiceBus). I'd like to discuss how proton uses this, what the spec actually says on how it should be used since its not clear to me these are currently in alignment, and possibly changing how we utilise it in proton. In Proton the outgoing-window is set based on the amount of outstanding outgoing bytes for the session (from buffered sends) and the frame size, which leads to many situations where the advertised value is 0, particularly in a client that creates sessions entirely independently from sending messages (such as a JMS client, or the Qpid Messaging C++ client). We then send transfer frames if a link has credit, regardless of the session outgoing window, only withholding them if the remote-incoming-window hits 0. This causes problems against ServiceBus because it seems to base its advertised incoming window on the clients [initial] advertised outgoing window. If it is 0, then it means we can never send any messages. The intent/definition of the outgoing-window seems a bit unclear as a whole in the spec to me, to the point you could almost remove it without issue given the [remote-]incoming-window seems to govern overall behaviour. However, it seems wrong to me that we often set the outgoing-window to 0. Doing this means we violate the outgoing window (local and at peer) whenever we send
