Re: put vs. send
Hah! I think I get it! Your comments about asynchronicity were the key. Rewriting now. - Original Message - On Tue, Mar 5, 2013 at 1:50 PM, Michael Goulish mgoul...@redhat.com wrote: - Original Message - On Tue, Mar 5, 2013 at 3:20 PM, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu rajit...@gmail.comwrote: On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross tr...@redhat.com wrote: On 03/05/2013 02:14 PM, Rajith Attapattu wrote: This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith Making this configurable multiplies the size of the test matrix. Can't we make this simpler? I do understand your concern here, but the Java impl already does both #1 and #2 and Rafi wants to do #3 in the future. The old JMS client does something similar. I agree that if we just do option #2 (as you suggest below), then the application can easily do #1 and #3 on top of that. But I'm sure they will like if the library implements those strategies for them and they have the ability to pick a strategy. I don't see why we'd make this configurable. All three options actually fit the same general semantics. Even if you're optimistically trying to transmit every single time put is called it's entirely possible for the socket to be blocked every single time you try. If this were to happen the implementation of #1 would appear to behave precisely the same as #2 behaves. In other words if you're coding correctly against the API you can't assume that put will or won't have transmitted anything regardless of which strategy is used internally. I agree with you. You make a very good point. Perhaps we should explicitly make that clear in our docs to avoid applications written against wrong assumptions. I can certainly do that, but it seems to me that semantics should be simple, obvious, and orthogonal. What seems non-simple and non-obvious to me so far is: put() might send, or not. It doesn't send now, but it might later. This behaviour is fundamental to an asynchronous API. You're not actually doing things, you're scheduling things to be done asynchronously. This is why put() returns a tracker so you can come back and check on the status of your asynchronous operation. recv() can cause messages to be sent. send() can cause messages to be received. I don't think that's a correct way of describing what is going on. You scheduled an asynchronous operation via put(). That means it can occur at any point later on. The fact that it happens to be trigger by the recv() in the example I gave is simply because recv() is blocking waiting for the reply and so it is inevitably going to end up blocking until the request is sent because the reply won't be triggered until after the request is sent. As for send(), it's simply inaccurate to say that send causes messages to be received. Messages can be spontaneously sent by remote parties at any time (given sufficient credit has been previously granted). What caused them to be received is the other party actually sending them, and if message data happens to arrived while we're inside a call to send(), we can't simply throw those messages away, so they go onto the incoming queue just as if they had arrived during a call to recv(). I would think that 1. every verb should only mean one thing 2. there should be a simple mental model, against which every verb performs a predictable action. so for example: put( messenger, message ); // enqueue for sending send ( messenger, BLOCK ); // block till all sent. send ( messenger, DONT_BLOCK ); // send what you can. credit ( messenger, 10 ); // limit incoming queue size recv ( messenger, BLOCK ); // block till I get a message recv ( messenger, DONT_BLOCK ); // if no messages incoming, return. I'm not
Re: put vs. send
On Wed, Mar 6, 2013 at 1:44 AM, Rob Godfrey rob.j.godf...@gmail.com wrote: On 5 March 2013 21:10, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross tr...@redhat.com wrote: [.. snip ..] It isn't really possible to have put cause messages to be eventually sent without a background thread, something we don't currently have. I think it's this that is what makes me find the API slightly odd. That put is an asynchronous operation is fine, but the fact that the only way to get work to occur is for a synchronous operation to be called seems a little screwy. If I understand correctly, right now an application programmer cannot actually write an asynchronous publisher, every so often they would have to call some form of synchronous operation. At the very least it would seem to suggest there might be call for a do some work but don't block function in the API. This could either take an aggressive strategy of flushing everything that it can to the wire, or it could attempt to optimize into larger transmission units. This is exactly what happens when you set the timeout to zero and call send (or recv). Are you saying you want some other way of doing the same thing or you want a background thread? --Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com wrote: This is exactly right. The API behaves in a surprising way and causes reasonable programmers to write programs that don't work. For the sake of adoption, we should fix this, not merely document it. This seems like a bit of a leap to me. Have we actually seen anyone misusing or abusing the API due to this? Mick didn't come across it till I pointed it out and even then he had to construct an experiment where he's basically observing the over-the-wire behaviour in order to detect it. --Rafael
Re: put vs. send
On 6 March 2013 13:26, Rafael Schloming r...@alum.mit.edu wrote: On Wed, Mar 6, 2013 at 1:44 AM, Rob Godfrey rob.j.godf...@gmail.com wrote: On 5 March 2013 21:10, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross tr...@redhat.com wrote: [.. snip ..] It isn't really possible to have put cause messages to be eventually sent without a background thread, something we don't currently have. I think it's this that is what makes me find the API slightly odd. That put is an asynchronous operation is fine, but the fact that the only way to get work to occur is for a synchronous operation to be called seems a little screwy. If I understand correctly, right now an application programmer cannot actually write an asynchronous publisher, every so often they would have to call some form of synchronous operation. At the very least it would seem to suggest there might be call for a do some work but don't block function in the API. This could either take an aggressive strategy of flushing everything that it can to the wire, or it could attempt to optimize into larger transmission units. This is exactly what happens when you set the timeout to zero and call send (or recv). Are you saying you want some other way of doing the same thing or you want a background thread? Surely though setting timeout to 0 and calling send results in something that looks like an error (this timed out). On a Java implementation I would expect this to throw an exception. That's not really the semantic I'm expecting. The semantic is do some work if you can without blocking. -- Rob --Rafael
[jira] [Created] (PROTON-266) JNIMessage does not encode message body
Keith Wall created PROTON-266: - Summary: JNIMessage does not encode message body Key: PROTON-266 URL: https://issues.apache.org/jira/browse/PROTON-266 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.4 Reporter: Keith Wall Fix For: 0.5 When using the Java bindings for Proton, currently calling #setBody() has no effect.It should be changed to update the _body thus allowing the message body to be encoded correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: put vs. send
Whether that's reported as an error is really a choice of the bindings. In C it's all just return codes. We could add a separate non-blocking flag that causes the blocking operations to return distinct error codes, i.e. the equivalent of EWOULDBLOCK, but I don't think this makes a whole lot of sense in C. I can buy that in the higher level bindings the extra flag would tell the API whether to signal timeout by returning false vs throwing an exception. With the Java implementation (not just the binding) I would expect an (expensive) exception to be thrown here. I don't think you should be triggering an exception for a non-exceptional condition. I do agree that we'll want a work interface at some point, but I've been thinking that would not just do the work, but also tell you what work has been done, so you can, e.g., go check whatever tracker statuses may have been updated. Yeah - i think what you are currently suggesting is more of a you can get round the lack of an explicit API because this sort of does the same thing if you squint at it. Calling a blocking method with a zero timeout is a hack to cover the lack of a method for the desired semantic. Moreover if this is a recommended use case for send then I think you'd need to document it, which would really muddy the waters as to what send is. -- Rob
Re: put vs. send
On Wed, Mar 6, 2013 at 6:37 AM, Rob Godfrey rob.j.godf...@gmail.com wrote: Whether that's reported as an error is really a choice of the bindings. In C it's all just return codes. We could add a separate non-blocking flag that causes the blocking operations to return distinct error codes, i.e. the equivalent of EWOULDBLOCK, but I don't think this makes a whole lot of sense in C. I can buy that in the higher level bindings the extra flag would tell the API whether to signal timeout by returning false vs throwing an exception. With the Java implementation (not just the binding) I would expect an (expensive) exception to be thrown here. I don't think you should be triggering an exception for a non-exceptional condition. How do you decide whether it's an exceptional condition or not? It seems like it's really down to how the app is designed as to whether timing out is normal or exceptional. I do agree that we'll want a work interface at some point, but I've been thinking that would not just do the work, but also tell you what work has been done, so you can, e.g., go check whatever tracker statuses may have been updated. Yeah - i think what you are currently suggesting is more of a you can get round the lack of an explicit API because this sort of does the same thing if you squint at it. Calling a blocking method with a zero timeout is a hack to cover the lack of a method for the desired semantic. Moreover if this is a recommended use case for send then I think you'd need to document it, which would really muddy the waters as to what send is. I'm suggesting it as a way to avoid adding a do_work() call because I'm not actually clear on how you would use the latter without busy looping or what scenarios you would document its use for. I'm not saying there aren't any, but it's not obvious to me right now. If you imagine the split between the non blocking and blocking portions of the API, where all the blocking portions are of the form do_work_and_block_until_condition_X_is_met, we now have two conditions: - the outgoing queue is empty - the incoming queue is non empty What you're asking for is to add a third condition that is always true, and I can possibly buy that for logical completeness, but in terms of usefulness I actually think expanding the set of conditions is actually more interesting, e.g. adding something like the outgoing queue is N perhaps via an optional parameter to send would seem to have a direct and obvious use for pipelined publishing in a way that wouldn't require busy looping. --Rafael
[jira] [Assigned] (PROTON-266) JNIMessage does not encode message body
[ https://issues.apache.org/jira/browse/PROTON-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wall reassigned PROTON-266: - Assignee: Keith Wall JNIMessage does not encode message body --- Key: PROTON-266 URL: https://issues.apache.org/jira/browse/PROTON-266 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.4 Reporter: Keith Wall Assignee: Keith Wall Fix For: 0.5 When using the Java bindings for Proton, currently calling #setBody() has no effect.It should be changed to update the _body thus allowing the message body to be encoded correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: put vs. send
On Wed, Mar 6, 2013 at 10:09 AM, Rafael Schloming r...@alum.mit.edu wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross tr...@redhat.com wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com wrote:
This is exactly right. The API behaves in a surprising way and causes
reasonable programmers to write programs that don't work. For the sake of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it till I
pointed it out and even then he had to construct an experiment where he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(**event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of events
goes quiet.
You can employ a timer along with your event count (or based on a byte
count) to get around that problem.
The timer will ensure you flush events when there isn't enough activity.
Isn't that acceptable ?
I guess I'm questioning the mission of the Messenger API. Which is the
more important design goal: general-purpose ease of use, or strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a really
nice thing to avoid if we can, and quite possibly a necessary mode of
operation in certain environments.
I don't think your example code will work though even if there is a
background thread.
This is a key point I missed when I thought about the problem along
the same lines as Ted.
Having a background thread cannot guarantee that your messages will be
written on to the wire as that thread can be blocked due to TCP
buffers being full or the thread being suppressed in favour of another
more higher priority thread (for longer than you desire) thus
increasing your latency beyond acceptable limits.
You will invariably have outliers in your latency graph.
On the other hand the library code will be much more simpler without
the background thread.
What do you want to happen when things start backing up?
Do you want messages to be dropped? Do you want put to start blocking? Do
you just want memory to grow indefinitely?
--Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 11:37 AM, Rajith Attapattu rajit...@gmail.com wrote:
On Wed, Mar 6, 2013 at 10:09 AM, Rafael Schloming r...@alum.mit.edu wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross tr...@redhat.com wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com wrote:
This is exactly right. The API behaves in a surprising way and causes
reasonable programmers to write programs that don't work. For the sake of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it till I
pointed it out and even then he had to construct an experiment where he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(**event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of events
goes quiet.
Having a background thread in the Messenger will only push this
problem from your application to the Messenger implementation.
Furthermore you will be at the mercy of the particulars of the client
library implementation as to how this background thread will take care
of the outstanding work.
We could provide all kinds of knobs to tweak and tune this behaviour,
but I'd be far more comfortable if I as the application developer can
be in control of when the flush happens.
Either way you will have arbitrarily/infinitely high latency due to
complications at the TCP stack or the OS level. But you can at least
help your case a bit by having the application issue the flush than
letting the messenger doing it, bcos the application is in a better
position to determine what are the optimal conditions for doing so and
those conditions could be other than time, msg or byte count.
You can employ a timer along with your event count (or based on a byte
count) to get around that problem.
The timer will ensure you flush events when there isn't enough activity.
Isn't that acceptable ?
I guess I'm questioning the mission of the Messenger API. Which is the
more important design goal: general-purpose ease of use, or strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a really
nice thing to avoid if we can, and quite possibly a necessary mode of
operation in certain environments.
I don't think your example code will work though even if there is a
background thread.
This is a key point I missed when I thought about the problem along
the same lines as Ted.
Having a background thread cannot guarantee that your messages will be
written on to the wire as that thread can be blocked due to TCP
buffers being full or the thread being suppressed in favour of another
more higher priority thread (for longer than you desire) thus
increasing your latency beyond acceptable limits.
You will invariably have outliers in your latency graph.
On the other hand the library code will be much more simpler without
the background thread.
What do you want to happen when things start backing up?
Do you want messages to be dropped? Do you want put to start blocking? Do
you just want memory to grow indefinitely?
--Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 9:01 AM, Michael Goulish mgoul...@redhat.com wrote:
- Original Message -
- Original Message -
From: Ted Ross tr...@redhat.com
To: proton@qpid.apache.org
Sent: Wednesday, March 6, 2013 10:35:47 AM
Subject: Re: put vs. send
On 03/06/2013 10:09 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross tr...@redhat.com
wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com
wrote:
This is exactly right. The API behaves in a surprising way
and
causes
reasonable programmers to write programs that don't work. For
the sake of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen
anyone
misusing or abusing the API due to this? Mick didn't come
across
it till I
pointed it out and even then he had to construct an experiment
where he's
basically observing the over-the-wire behaviour in order to
detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(**event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum
message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of
events
goes quiet.
I guess I'm questioning the mission of the Messenger API. Which
is the
more important design goal: general-purpose ease of use, or
strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a
really
nice thing to avoid if we can, and quite possibly a necessary
mode
of
operation in certain environments.
I don't think your example code will work though even if there is
a
background thread. What do you want to happen when things start
backing up?
Do you want messages to be dropped? Do you want put to start
blocking? Do
you just want memory to grow indefinitely?
Good question. I would want to either block so I can stop
consuming
events or get an indication that I would-block so I can take other
actions. I understand that this is what send is for, but it's
not
clear when, or how often I should call it.
This begs a question that was asked before (by Mick, I believe) -
what happens if a put() message can _never_ be sent? The
destination has gone away and will never come back. AFAIK, every
further call to send() will block due to that stuck message. How
should that be dealt with? Use a TTL?
Well, I just tried it.
Setup
---
Set up a receiver to receive, get, and print out messages in a loop.
( receive blocking, i.e. timeout == default )
Then start a sender (default timeout) that will:
1. put-and-send message 1.
2. put message 2.
3. countdown 12 seconds before it decides to send message 2.
4. send message 2.
While the sender is getting ready to call send() on msg2, I kill the
receiver.
Result
---
I see the receiver print out message 1. good.
When the sender has put() msg2 but not yet sent(), I kill receiver.
Sender calls send() on message2.
send() returns immediately, return code is 0. (success)
Analysis
--
Bummer. Dropped message.
If you want reliability you need to check the status of the tracker you get
when you call put(). You would also need to set a nonzero outgoing window
so that messenger actually retains that status.
--Rafael
