Re: Fwd: put vs. send
On 3/8/13 3:19 PM, Rafael Schloming wrote: On Thu, Mar 7, 2013 at 5:15 AM, Bozo Dragojevic bo...@digiverse.si mailto:bo...@digiverse.si wrote: On 3/6/13 3:45 AM, Rafael Schloming wrote: Oops, meant to send this to the list. -- Forwarded message -- From: Rafael Schlomingr...@alum.mit.edu mailto:r...@alum.mit.edu Date: Tue, Mar 5, 2013 at 6:44 PM Subject: Re: put vs. send To: Bozo Dragojevicbo...@digiverse.si mailto:bo...@digiverse.si [snip] Thank you for taking the time to explain, that's very helpful. It makes me wonder if we shouldn't include some kind of pn_messenger_interrupt() that to pull out of blocking calls early. That'd be great! Few weeks ago I've submitted https://issues.apache.org/jira/browse/PROTON-231 for this feature and the JIRA has a link to my attempt at implementation (based off 0.3 branch) -- this is what we use now. I've named it pn_messenger_wakeup() after pn_driver_wakeup() that it calls. In addition the pn_messenger_tsync() needed to be adjusted to actually exit early :) So for me this opens up two new areas (one of the reasons it took me a bit to reply): - study up the API on 'trackers', iiuc thats the name of what pn_messenger_put() returns - figure out how to use them inside our code so that the publisher (API user) will not drown itself in messages it cannot realistically send. I'm not sure trackers are what you want here (although it may be depending on your app). Trackers are used to track the status of the message at the receiver, e.g. it will tell you whether the receiver has accepted or rejected the message. the fourth paragraph below circles back to trackers :) If you just care about limiting the rate of your producer to match what you can actually push onto the wire, then you can just look at pn_messenger_outgoing(), that will tell you how many messages are actually backed up waiting to go onto the wire, e.g. before or after doing a put/send/recv/etc, you can check the size of the outgoing queue and based on that decide whether you need to throttle your producer. Looking at total size of outgoing queue is sufficient for the messenger that is part of the application that acts as the publisher and sends messages to the broker as it then has just one peer. The scenario that is important is this: Broker, too, wants to rate-limit each peer based solely on that peer's capability to receive messages. So if there are two subscribers and one is on a slower link, or is not behaving in some other way, then the fast subscriber should not be held hostage of the slow subscriber. This is even more so if we're talking about two publishers each having a set of own subscribers. On the messenger level, the broker ends up having a bunch of amqp addresses where the message received from the publisher need to be forwarded. The answer that is sought is pn_messenger_outgoing(address). Now, a tracker does uniquely identify the remote address that is embedded in the message being sent. If there is/were a query of local messenger's opinion on progress status wrt. delivering of that one message where one distinct answer would be managed to put it on the wire. A function like that makes it trivial for a messenger user (broker) to perform the accounting of 'number of outstanding messages per address' on it's own. - ability to spread out load of serialization and deserialization over cores and as mentioned above there are two 'layers' of it: bytes = pn_message = API object Providing this within the messenger paradigm sounds tricky to me -- I definitely don't want messenger to grow it's own thread pool :) This sounds to me like it could possibly be addressed by a smarter pn_message implementation. Right now pn_message_t is implemented in a very basic way where it actually does a lot of unnecessary encode/decode work. Not only could that be made more efficient so there is much less load to be distributed, but it could also be made to happen on demand. That would give you the flexibility of hitting most of the cost of the bytes = pn_message_t portion in whichever thread you chose. This multi-core business obviously makes most sense for cases where there is more than one TCP connection where data flows in. The thread that read a bunch of bytes from a socket already has them in that CPU cache. Doing as much of the work with those bytes on that CPU, without context switches, will be fast. - more control over IO and waiting (as per description above) this alone would still fit into your proposed extension quite nicely In the other thread you mention bringing send() and recv() closer wrt meaning of a parameter. It'd be a tiny step from there to provide a sync(send_limit, recv_limit) which would help me in reasoning of there being exactly
Re: Fwd: put vs. send
On Thu, Mar 7, 2013 at 5:15 AM, Bozo Dragojevic bo...@digiverse.si wrote:
On 3/6/13 3:45 AM, Rafael Schloming wrote:
Oops, meant to send this to the list.
-- Forwarded message --
From: Rafael Schlomingr...@alum.mit.edu
Date: Tue, Mar 5, 2013 at 6:44 PM
Subject: Re: put vs. send
To: Bozo Dragojevicbo...@digiverse.si
On Tue, Mar 5, 2013 at 3:25 PM, Bozo Dragojevicbo...@digiverse.si
wrote:
Wow, by not ever calling pn_messenger_send(), but only
pn_messenger_recv()
things are, unexpectedly, better! I'll explain below what 'better' means.
But first, this begs the question, what is the purpose of
pn_messenger_send()
and where (and why) it's appropriate/required to call it.
Well one example would be if you just want to send a bunch of messages
without ever receiving any.
I think it helps if you conceptually split the API along
blocking/non-blocking lines:
non-blocking: put, subscribe, get, incoming, outgoing, etc...
blocking: send, recv
Of the non blocking portion there are things that simply access data, like
get(), incoming, outgoing, etc, and then there are things that
schedule/request asynchronous work to be done, namely put() and
subscribe().
For the blocking portion, I think it helps to understand that there is
really only one fundamental operation, and that is
do_any_outstanding_work_until_**condition_X_is_met. Currently the
blocking
portion of the API exposes exactly two conditions, namely all messages
are
sent via send() and there is a message waiting to be received via
recv().
So in your example below if you're basically just a request/response
server
then as you've noticed you never actually need to call send since you
don't
really care about waiting for that condition, and any work you schedule in
your handlers will be performed anyways as you wait for messages to be
received.
Thanks for this really good explanation. This plus your proposed change in
the other part of the thread would make the API much more
The results are for slightly modified 0.3 release.
Most notably, I have a local change that exposes pn_driver_wakeup()
through the messenger api.
I'm curious about why you had to make this change. Can you describe the
usage/scenario that requires it?
Our design has a 'protocol' thread that is separate from the API,
the idea being we can already be processing next incoming network
messages while the API user is still busy servicing the previous one.
By processing I mean all the CPU-required deserialization work of
converting the network packets first to pn_message() and then to our
internal higher-level representation of the message content.
The mechanism how API thread sends a message is to hand off such a
high-level objects off to the 'proto' thread via a queue.
so we end up with a (proto) thread that has two wakeup sources
a) network related events (ability to send more stuff or receive it), and
b) commands from the API coming from the queue.
Idle state for the proto thread is no network activity and no API
activity and in that state we want to be blocked somewhere. We choose
pn_driver_wait() as it's a natural wait point and also we can have best
possible incoming message latency.
We also need to react quickly to messages that API wants to send out.
Given the messenger is hiding it's driver it seemed a smaller change
to expose the pn_driver_wakeup() via messenger (and teach messenger
to actually come out of it's tsync method).
Even if we did add locking and would be able to call pn_messenger_put()
from a separate thread, while we left the 'proto' thread blocked in
pn_driver_wait() it wouldn't help as the poll mask would be wrong --
if we were not sending before then pn_driver_rebuild() had not selected
writable condition for that connection -- or we wouldn't be blocked in
pn_driver_wait() -- talking about a mostly-idle low latency situation
where TCP buffers are not clogged up.
hope this clarifies things more than it muddles them up :)
Thank you for taking the time to explain, that's very helpful. It makes me
wonder if we shouldn't include some kind of pn_messenger_interrupt() that
to pull out of blocking calls early.
Our API is threaded internally but all proton communication is done
via a dedicated thread that runs the following (stub) event loop:
(The same event loop is used by both, the client library, and the
'broker')
while(1) {
ret = pn_messenger_recv(m,100) // the 100 is hard to explain...
if (ret != PN_WAKED_UP) {// new error code for wakeup case
/*
* apparently there's no need to call send...
* pn_messenger_send(m);
*/
}
Command cmd = cmd_queue.get(); // cmd_queue.put() from another
thread
// will call pn_driver_wakeup() and
will
// break out of pn_messenger_recv()
if (cmd)
handle
Re: put vs. send
- Original Message -
From: Rafael Schloming r...@alum.mit.edu
To: proton@qpid.apache.org
Sent: Wednesday, March 6, 2013 2:27:19 PM
Subject: Re: put vs. send
On Wed, Mar 6, 2013 at 7:35 AM, Ted Ross tr...@redhat.com wrote:
On 03/06/2013 10:09 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross tr...@redhat.com wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com
wrote:
This is exactly right. The API behaves in a surprising way
and causes
reasonable programmers to write programs that don't work. For
the sake
of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen
anyone
misusing or abusing the API due to this? Mick didn't come across
it
till I
pointed it out and even then he had to construct an experiment
where
he's
basically observing the over-the-wire behaviour in order to
detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum
message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of
events
goes quiet.
I guess I'm questioning the mission of the Messenger API. Which
is the
more important design goal: general-purpose ease of use, or
strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a
really
nice thing to avoid if we can, and quite possibly a necessary mode
of
operation in certain environments.
I don't think your example code will work though even if there is
a
background thread. What do you want to happen when things start
backing
up?
Do you want messages to be dropped? Do you want put to start
blocking? Do
you just want memory to grow indefinitely?
Good question. I would want to either block so I can stop
consuming
events or get an indication that I would-block so I can take other
actions.
I understand that this is what send is for, but it's not clear
when, or
how often I should call it.
I think there are really two orthogonal issues being discussed:
1) how do you get messenger to perform the appropriate amount of
outstanding work once enough has built up
2) should messenger be obligated to eventually complete any
outstanding
work even if you never call into it again
The second one basically being the question of whether the API
semantics
mandate a background thread or not, and I think (1) needs a good
solution
regardless of how you answer (2).
I think we can address (1) by adding an integer argument N to
pn_messenger_send with the following semantics:
- if N is -1, then pn_messenger_send behaves as it currently does,
i.e.
block until everything is sent
- if N is 0, then pn_messenger_send will send whatever it
can/should
without blocking
- if N is positive, then pn_messenger_send will block until N
messages
are sent
I'd also suggest we modify pn_messenger_recv(0) to match this on the
recv
side, so the overall semantics of pn_messenger_recv(N) would be:
- if N is -1, then pn_messenger_recv behaves as it currently does,
i.e.
block until something is received
- if N is 0, then pn_messenger_recv will receive whatever it
can/should
without blocking
- if N is positive, then pn_messenger_recv behaves as it currently
does,
i.e. block until something is received but with a limit of N
+1 these proposed API changes.
I like the fact that they provide more control over the blocking behavior.
With the existing API, I've had to modify pn_messenger_set/get_timeout() if I
want to switch between polling and blocking. The proposed approach is much
cleaner.
I think the above change would introduce some symmetry between send
and
recv and let you implement a fully pipelined asynchronous sender in a
more
obvious way than setting timeouts to zero, e.g. Ted's example could
read:
while (True) {
wait_for_and_get_next_event(**event);
pn_messenger_put(m, event);
if (pn_messenger_outgoing() threshold) {
pn_messenger_send(m, pn_messenger_outgoing() - threshold);
}
}
We can also add a pn_messenger_work(timeout) that would block until
it's
done some kind of work (either sending or receiving). Calling it with
a
zero argument would be equivalent to simultaneously calling
pn_messenger_send(0) and pn_messenger_recv(0). (Note that currently
send
and recv don't actually factor in any biases towards sending and
receiving,
they just do whatever outstanding work needs doing, however the API
would
admit an implementation that did apply some bias in those cases.)
This would go some ways towards addressing issue (2) because
Re: put vs. send -- new doc
I like this. I don't think this is the same thing as high level conceptual intro, but may well be more useful right now, and it will always be very useful as recipe book style documentation. I think this is actually quite helpful for the ongoing API discussions. One of the tricky things about developing a simple API is that everyone has their own scenario that they want to be simple, and sometimes making one scenario simpler ends up making another one more difficult, so it's actually really useful to have them all laid out so concisely so that we can observe the overall effect of any changes. I'm thinking we should expand this with the API changes I proposed in the other thread and see how they work out. --Rafael On Wed, Mar 6, 2013 at 9:43 AM, Michael Goulish mgoul...@redhat.com wrote: OK, I'm trying here to express the spirit of Messenger I/O , greatly based on the conversation of the last 24 hrs. This probably needs some elaboration yet, but I want to see if I'm at least generally on the right track. Oh, please, give me feedback. Sending and Receiving Messages === The Proton Messenger API provides a mixture of synchronous and asynchronous operations to give you flexibility in deciding when you application should block waiting for I/O, and when it should not. When sending messages, you can: * send a message immediately, * enqueue a message to be sent later, * block until all enqueued messages are sent, * send enqueued messages until a timeout occurs, or * send all messages that can be sent without blocking. When receiving messages, you can: * receive messages that can be received without blocking, * block until at least one message is received, * receive no more than a fixed number of messages. Examples -- 1. send a message immediately put ( messenger, msg ); send ( messenger ); 2. enqueue a message to be sent later put ( messenger, msg ); note: The message will be sent whenever it is not blocked and the Messenger code has other I/O work to be done. 3. block until all enqueued messages are sent set_timeout ( messenger, -1 ); send ( messenger ); note: A negative timeout means 'forever'. That is the initial default for a messenger. 4. send enqueued messages until a timeout occurs set_timeout ( messenger, 100 ); /* 100 msec */ send ( messenger ); 5. send all messages that can be sent without blocking set_timeout ( messenger, 0 ); send ( messenger ); 6. receive messages that can be received without blocking set_timeout ( messenger, 0 ); recv ( messenger, -1 ); 7. block until at least one message is received set_timeout ( messenger, -1 ); recv ( messenger, -1 ); note: -1 is initial messenger default. If you didn't change it, you don't need to set it. 8. receive no more than a fixed number of messages recv ( messenger, 10 );
Re: put vs. send -- new doc
- Original Message - I like this. Good! I'm trying to get at the intention, as I understood it from online discussions, and make a doc that makes the intention easy to see. I don't think this is the same thing as high level conceptual intro, but may well be more useful right now, and it will always be very useful as recipe book style documentation. No, this isn't the intro -- I'm re-working that now. I guess I should have said what category of docs this is part of. Except I don't actually know. I guess I have an idea of a set of docs at this level that discuss different topics. i.e. addressing, message management, etc. I think this is actually quite helpful for the ongoing API discussions. One of the tricky things about developing a simple API is that everyone has their own scenario that they want to be simple, and sometimes making one scenario simpler ends up making another one more difficult, so it's actually really useful to have them all laid out so concisely so that we can observe the overall effect of any changes. I'm thinking we should expand this with the API changes I proposed in the other thread and see how they work out. Sure -- like a doc for your recent proposals for changes. That would be fun, and useful I agree. But -- probably shouldn't check that into tree, right? Since it's for hypothetical code. Just post to list ? --Rafael On Wed, Mar 6, 2013 at 9:43 AM, Michael Goulish mgoul...@redhat.com wrote: OK, I'm trying here to express the spirit of Messenger I/O , greatly based on the conversation of the last 24 hrs. This probably needs some elaboration yet, but I want to see if I'm at least generally on the right track. Oh, please, give me feedback. Sending and Receiving Messages === The Proton Messenger API provides a mixture of synchronous and asynchronous operations to give you flexibility in deciding when you application should block waiting for I/O, and when it should not. When sending messages, you can: * send a message immediately, * enqueue a message to be sent later, * block until all enqueued messages are sent, * send enqueued messages until a timeout occurs, or * send all messages that can be sent without blocking. When receiving messages, you can: * receive messages that can be received without blocking, * block until at least one message is received, * receive no more than a fixed number of messages. Examples -- 1. send a message immediately put ( messenger, msg ); send ( messenger ); 2. enqueue a message to be sent later put ( messenger, msg ); note: The message will be sent whenever it is not blocked and the Messenger code has other I/O work to be done. 3. block until all enqueued messages are sent set_timeout ( messenger, -1 ); send ( messenger ); note: A negative timeout means 'forever'. That is the initial default for a messenger. 4. send enqueued messages until a timeout occurs set_timeout ( messenger, 100 ); /* 100 msec */ send ( messenger ); 5. send all messages that can be sent without blocking set_timeout ( messenger, 0 ); send ( messenger ); 6. receive messages that can be received without blocking set_timeout ( messenger, 0 ); recv ( messenger, -1 ); 7. block until at least one message is received set_timeout ( messenger, -1 ); recv ( messenger, -1 ); note: -1 is initial messenger default. If you didn't change it, you don't need to set it. 8. receive no more than a fixed number of messages recv ( messenger, 10 );
Re: put vs. send -- new doc
On Thu, Mar 7, 2013 at 3:18 PM, Michael Goulish mgoul...@redhat.com wrote: I think this is actually quite helpful for the ongoing API discussions. One of the tricky things about developing a simple API is that everyone has their own scenario that they want to be simple, and sometimes making one scenario simpler ends up making another one more difficult, so it's actually really useful to have them all laid out so concisely so that we can observe the overall effect of any changes. I'm thinking we should expand this with the API changes I proposed in the other thread and see how they work out. Sure -- like a doc for your recent proposals for changes. That would be fun, and useful I agree. But -- probably shouldn't check that into tree, right? Since it's for hypothetical code. Just post to list ? Right, I wouldn't put it in the tree, just go for the list right now. --Rafael
Re: put vs. send
Hah! I think I get it! Your comments about asynchronicity were the key. Rewriting now. - Original Message - On Tue, Mar 5, 2013 at 1:50 PM, Michael Goulish mgoul...@redhat.com wrote: - Original Message - On Tue, Mar 5, 2013 at 3:20 PM, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu rajit...@gmail.comwrote: On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross tr...@redhat.com wrote: On 03/05/2013 02:14 PM, Rajith Attapattu wrote: This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith Making this configurable multiplies the size of the test matrix. Can't we make this simpler? I do understand your concern here, but the Java impl already does both #1 and #2 and Rafi wants to do #3 in the future. The old JMS client does something similar. I agree that if we just do option #2 (as you suggest below), then the application can easily do #1 and #3 on top of that. But I'm sure they will like if the library implements those strategies for them and they have the ability to pick a strategy. I don't see why we'd make this configurable. All three options actually fit the same general semantics. Even if you're optimistically trying to transmit every single time put is called it's entirely possible for the socket to be blocked every single time you try. If this were to happen the implementation of #1 would appear to behave precisely the same as #2 behaves. In other words if you're coding correctly against the API you can't assume that put will or won't have transmitted anything regardless of which strategy is used internally. I agree with you. You make a very good point. Perhaps we should explicitly make that clear in our docs to avoid applications written against wrong assumptions. I can certainly do that, but it seems to me that semantics should be simple, obvious, and orthogonal. What seems non-simple and non-obvious to me so far is: put() might send, or not. It doesn't send now, but it might later. This behaviour is fundamental to an asynchronous API. You're not actually doing things, you're scheduling things to be done asynchronously. This is why put() returns a tracker so you can come back and check on the status of your asynchronous operation. recv() can cause messages to be sent. send() can cause messages to be received. I don't think that's a correct way of describing what is going on. You scheduled an asynchronous operation via put(). That means it can occur at any point later on. The fact that it happens to be trigger by the recv() in the example I gave is simply because recv() is blocking waiting for the reply and so it is inevitably going to end up blocking until the request is sent because the reply won't be triggered until after the request is sent. As for send(), it's simply inaccurate to say that send causes messages to be received. Messages can be spontaneously sent by remote parties at any time (given sufficient credit has been previously granted). What caused them to be received is the other party actually sending them, and if message data happens to arrived while we're inside a call to send(), we can't simply throw those messages away, so they go onto the incoming queue just as if they had arrived during a call to recv(). I would think that 1. every verb should only mean one thing 2. there should be a simple mental model, against which every verb performs a predictable action. so for example: put( messenger, message ); // enqueue for sending send ( messenger, BLOCK ); // block till all sent. send ( messenger, DONT_BLOCK ); // send what you can. credit ( messenger, 10 ); // limit incoming queue size recv ( messenger, BLOCK ); // block till I get a message recv ( messenger, DONT_BLOCK ); // if no messages incoming, return. I'm not
Re: put vs. send
On Wed, Mar 6, 2013 at 1:44 AM, Rob Godfrey rob.j.godf...@gmail.com wrote: On 5 March 2013 21:10, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross tr...@redhat.com wrote: [.. snip ..] It isn't really possible to have put cause messages to be eventually sent without a background thread, something we don't currently have. I think it's this that is what makes me find the API slightly odd. That put is an asynchronous operation is fine, but the fact that the only way to get work to occur is for a synchronous operation to be called seems a little screwy. If I understand correctly, right now an application programmer cannot actually write an asynchronous publisher, every so often they would have to call some form of synchronous operation. At the very least it would seem to suggest there might be call for a do some work but don't block function in the API. This could either take an aggressive strategy of flushing everything that it can to the wire, or it could attempt to optimize into larger transmission units. This is exactly what happens when you set the timeout to zero and call send (or recv). Are you saying you want some other way of doing the same thing or you want a background thread? --Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com wrote: This is exactly right. The API behaves in a surprising way and causes reasonable programmers to write programs that don't work. For the sake of adoption, we should fix this, not merely document it. This seems like a bit of a leap to me. Have we actually seen anyone misusing or abusing the API due to this? Mick didn't come across it till I pointed it out and even then he had to construct an experiment where he's basically observing the over-the-wire behaviour in order to detect it. --Rafael
Re: put vs. send
On 6 March 2013 13:26, Rafael Schloming r...@alum.mit.edu wrote: On Wed, Mar 6, 2013 at 1:44 AM, Rob Godfrey rob.j.godf...@gmail.com wrote: On 5 March 2013 21:10, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross tr...@redhat.com wrote: [.. snip ..] It isn't really possible to have put cause messages to be eventually sent without a background thread, something we don't currently have. I think it's this that is what makes me find the API slightly odd. That put is an asynchronous operation is fine, but the fact that the only way to get work to occur is for a synchronous operation to be called seems a little screwy. If I understand correctly, right now an application programmer cannot actually write an asynchronous publisher, every so often they would have to call some form of synchronous operation. At the very least it would seem to suggest there might be call for a do some work but don't block function in the API. This could either take an aggressive strategy of flushing everything that it can to the wire, or it could attempt to optimize into larger transmission units. This is exactly what happens when you set the timeout to zero and call send (or recv). Are you saying you want some other way of doing the same thing or you want a background thread? Surely though setting timeout to 0 and calling send results in something that looks like an error (this timed out). On a Java implementation I would expect this to throw an exception. That's not really the semantic I'm expecting. The semantic is do some work if you can without blocking. -- Rob --Rafael
Re: put vs. send
Whether that's reported as an error is really a choice of the bindings. In C it's all just return codes. We could add a separate non-blocking flag that causes the blocking operations to return distinct error codes, i.e. the equivalent of EWOULDBLOCK, but I don't think this makes a whole lot of sense in C. I can buy that in the higher level bindings the extra flag would tell the API whether to signal timeout by returning false vs throwing an exception. With the Java implementation (not just the binding) I would expect an (expensive) exception to be thrown here. I don't think you should be triggering an exception for a non-exceptional condition. I do agree that we'll want a work interface at some point, but I've been thinking that would not just do the work, but also tell you what work has been done, so you can, e.g., go check whatever tracker statuses may have been updated. Yeah - i think what you are currently suggesting is more of a you can get round the lack of an explicit API because this sort of does the same thing if you squint at it. Calling a blocking method with a zero timeout is a hack to cover the lack of a method for the desired semantic. Moreover if this is a recommended use case for send then I think you'd need to document it, which would really muddy the waters as to what send is. -- Rob
Re: put vs. send
On Wed, Mar 6, 2013 at 6:37 AM, Rob Godfrey rob.j.godf...@gmail.com wrote: Whether that's reported as an error is really a choice of the bindings. In C it's all just return codes. We could add a separate non-blocking flag that causes the blocking operations to return distinct error codes, i.e. the equivalent of EWOULDBLOCK, but I don't think this makes a whole lot of sense in C. I can buy that in the higher level bindings the extra flag would tell the API whether to signal timeout by returning false vs throwing an exception. With the Java implementation (not just the binding) I would expect an (expensive) exception to be thrown here. I don't think you should be triggering an exception for a non-exceptional condition. How do you decide whether it's an exceptional condition or not? It seems like it's really down to how the app is designed as to whether timing out is normal or exceptional. I do agree that we'll want a work interface at some point, but I've been thinking that would not just do the work, but also tell you what work has been done, so you can, e.g., go check whatever tracker statuses may have been updated. Yeah - i think what you are currently suggesting is more of a you can get round the lack of an explicit API because this sort of does the same thing if you squint at it. Calling a blocking method with a zero timeout is a hack to cover the lack of a method for the desired semantic. Moreover if this is a recommended use case for send then I think you'd need to document it, which would really muddy the waters as to what send is. I'm suggesting it as a way to avoid adding a do_work() call because I'm not actually clear on how you would use the latter without busy looping or what scenarios you would document its use for. I'm not saying there aren't any, but it's not obvious to me right now. If you imagine the split between the non blocking and blocking portions of the API, where all the blocking portions are of the form do_work_and_block_until_condition_X_is_met, we now have two conditions: - the outgoing queue is empty - the incoming queue is non empty What you're asking for is to add a third condition that is always true, and I can possibly buy that for logical completeness, but in terms of usefulness I actually think expanding the set of conditions is actually more interesting, e.g. adding something like the outgoing queue is N perhaps via an optional parameter to send would seem to have a direct and obvious use for pipelined publishing in a way that wouldn't require busy looping. --Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 10:09 AM, Rafael Schloming r...@alum.mit.edu wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross tr...@redhat.com wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com wrote:
This is exactly right. The API behaves in a surprising way and causes
reasonable programmers to write programs that don't work. For the sake of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it till I
pointed it out and even then he had to construct an experiment where he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(**event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of events
goes quiet.
You can employ a timer along with your event count (or based on a byte
count) to get around that problem.
The timer will ensure you flush events when there isn't enough activity.
Isn't that acceptable ?
I guess I'm questioning the mission of the Messenger API. Which is the
more important design goal: general-purpose ease of use, or strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a really
nice thing to avoid if we can, and quite possibly a necessary mode of
operation in certain environments.
I don't think your example code will work though even if there is a
background thread.
This is a key point I missed when I thought about the problem along
the same lines as Ted.
Having a background thread cannot guarantee that your messages will be
written on to the wire as that thread can be blocked due to TCP
buffers being full or the thread being suppressed in favour of another
more higher priority thread (for longer than you desire) thus
increasing your latency beyond acceptable limits.
You will invariably have outliers in your latency graph.
On the other hand the library code will be much more simpler without
the background thread.
What do you want to happen when things start backing up?
Do you want messages to be dropped? Do you want put to start blocking? Do
you just want memory to grow indefinitely?
--Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 11:37 AM, Rajith Attapattu rajit...@gmail.com wrote:
On Wed, Mar 6, 2013 at 10:09 AM, Rafael Schloming r...@alum.mit.edu wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross tr...@redhat.com wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com wrote:
This is exactly right. The API behaves in a surprising way and causes
reasonable programmers to write programs that don't work. For the sake of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it till I
pointed it out and even then he had to construct an experiment where he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(**event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of events
goes quiet.
Having a background thread in the Messenger will only push this
problem from your application to the Messenger implementation.
Furthermore you will be at the mercy of the particulars of the client
library implementation as to how this background thread will take care
of the outstanding work.
We could provide all kinds of knobs to tweak and tune this behaviour,
but I'd be far more comfortable if I as the application developer can
be in control of when the flush happens.
Either way you will have arbitrarily/infinitely high latency due to
complications at the TCP stack or the OS level. But you can at least
help your case a bit by having the application issue the flush than
letting the messenger doing it, bcos the application is in a better
position to determine what are the optimal conditions for doing so and
those conditions could be other than time, msg or byte count.
You can employ a timer along with your event count (or based on a byte
count) to get around that problem.
The timer will ensure you flush events when there isn't enough activity.
Isn't that acceptable ?
I guess I'm questioning the mission of the Messenger API. Which is the
more important design goal: general-purpose ease of use, or strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a really
nice thing to avoid if we can, and quite possibly a necessary mode of
operation in certain environments.
I don't think your example code will work though even if there is a
background thread.
This is a key point I missed when I thought about the problem along
the same lines as Ted.
Having a background thread cannot guarantee that your messages will be
written on to the wire as that thread can be blocked due to TCP
buffers being full or the thread being suppressed in favour of another
more higher priority thread (for longer than you desire) thus
increasing your latency beyond acceptable limits.
You will invariably have outliers in your latency graph.
On the other hand the library code will be much more simpler without
the background thread.
What do you want to happen when things start backing up?
Do you want messages to be dropped? Do you want put to start blocking? Do
you just want memory to grow indefinitely?
--Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 9:01 AM, Michael Goulish mgoul...@redhat.com wrote:
- Original Message -
- Original Message -
From: Ted Ross tr...@redhat.com
To: proton@qpid.apache.org
Sent: Wednesday, March 6, 2013 10:35:47 AM
Subject: Re: put vs. send
On 03/06/2013 10:09 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross tr...@redhat.com
wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross tr...@redhat.com
wrote:
This is exactly right. The API behaves in a surprising way
and
causes
reasonable programmers to write programs that don't work. For
the sake of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen
anyone
misusing or abusing the API due to this? Mick didn't come
across
it till I
pointed it out and even then he had to construct an experiment
where he's
basically observing the over-the-wire behaviour in order to
detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(**event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum
message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of
events
goes quiet.
I guess I'm questioning the mission of the Messenger API. Which
is the
more important design goal: general-purpose ease of use, or
strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a
really
nice thing to avoid if we can, and quite possibly a necessary
mode
of
operation in certain environments.
I don't think your example code will work though even if there is
a
background thread. What do you want to happen when things start
backing up?
Do you want messages to be dropped? Do you want put to start
blocking? Do
you just want memory to grow indefinitely?
Good question. I would want to either block so I can stop
consuming
events or get an indication that I would-block so I can take other
actions. I understand that this is what send is for, but it's
not
clear when, or how often I should call it.
This begs a question that was asked before (by Mick, I believe) -
what happens if a put() message can _never_ be sent? The
destination has gone away and will never come back. AFAIK, every
further call to send() will block due to that stuck message. How
should that be dealt with? Use a TTL?
Well, I just tried it.
Setup
---
Set up a receiver to receive, get, and print out messages in a loop.
( receive blocking, i.e. timeout == default )
Then start a sender (default timeout) that will:
1. put-and-send message 1.
2. put message 2.
3. countdown 12 seconds before it decides to send message 2.
4. send message 2.
While the sender is getting ready to call send() on msg2, I kill the
receiver.
Result
---
I see the receiver print out message 1. good.
When the sender has put() msg2 but not yet sent(), I kill receiver.
Sender calls send() on message2.
send() returns immediately, return code is 0. (success)
Analysis
--
Bummer. Dropped message.
If you want reliability you need to check the status of the tracker you get
when you call put(). You would also need to set a nonzero outgoing window
so that messenger actually retains that status.
--Rafael
put vs. send
quoth Rafi: The semantics of pn_messenger_put allow it to send if it can do so without blocking. So, am I understanding correctly? -- I should be able to get messages from my sender to my receiver just by calling put() -- if the receiver is ready to receive? The only transmission difference between put() and send() is that send() will actually block until they're all sent (or timeout hits). put() should get rid of all the messages that aren't blocked, and leave all that are. . . . Because what I'm seeing is -- with my receiver hanging in recv(), I put 5 messages. Sender sits there for a while. No messages arrive at receiver. Then sender calls send() -- and all 5 messages arrive at the receiver. This is true whether on the receiver side, I use pn_messenger_recv ( messenger, 100 ); pn_messenger_recv ( messenger, 5 ); pn_messenger_recv ( messenger, 1 ); or pn_messenger_recv ( messenger, -1 ); That's why it seemed two-stage to me. put() seems to gets them staged, send() seems to shove them out the door. No? Or is this a bug?
Re: put vs. send
On 03/05/2013 02:01 PM, Rafael Schloming wrote: On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish mgoul...@redhat.comwrote: So, am I understanding correctly? -- I should be able to get messages from my sender to my receiver just by calling put() -- if the receiver is ready to receive? Not necessarily, the receiver being ready just means you are unblocked on AMQP level flow control. You could also potentially block on the socket write (i.e. TCP level flow control). You need to be unblocked on both for put to succeed. Certainly there is no TCP flow control happening in Mick's scenario. What I said was put is *allowed* to send optimistically, not that it is required to. It actually did send optimistically in a previous version of the code, however I commented that line out. I would say the documented semantics of put and send should allow the implementation the flexibility to do any of the following: 1) optimistically transmit whatever it can everytime so long as it doesn't block 2) never bother transmitting anything until you force it to by calling send 3) anything in between the first two, e.g. magically transmit once you've put enough messages to reach the optimal batch size The reason for the behaviour you are observing is that we currently do option 2 in the C impl, however we've done option 1 in the past (and I think we do option 1 still in the Java impl), and we will probably do option 3 in the future. If this is the case, then Mick's original view is correct. The application must assume that messages will not ever be sent unless send is called. There is no flowing, pipelined, non-blocking producer. --Rafael
Re: put vs. send
On Tue, Mar 5, 2013 at 2:01 PM, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish mgoul...@redhat.comwrote: quoth Rafi: The semantics of pn_messenger_put allow it to send if it can do so without blocking. So, am I understanding correctly? -- I should be able to get messages from my sender to my receiver just by calling put() -- if the receiver is ready to receive? Not necessarily, the receiver being ready just means you are unblocked on AMQP level flow control. You could also potentially block on the socket write (i.e. TCP level flow control). You need to be unblocked on both for put to succeed. The only transmission difference between put() and send() is that send() will actually block until they're all sent (or timeout hits). put() should get rid of all the messages that aren't blocked, and leave all that are. . . . Because what I'm seeing is -- with my receiver hanging in recv(), I put 5 messages. Sender sits there for a while. No messages arrive at receiver. Then sender calls send() -- and all 5 messages arrive at the receiver. This is true whether on the receiver side, I use pn_messenger_recv ( messenger, 100 ); pn_messenger_recv ( messenger, 5 ); pn_messenger_recv ( messenger, 1 ); or pn_messenger_recv ( messenger, -1 ); That's why it seemed two-stage to me. put() seems to gets them staged, send() seems to shove them out the door. No? Or is this a bug? What I said was put is *allowed* to send optimistically, not that it is required to. It actually did send optimistically in a previous version of the code, however I commented that line out. I would say the documented semantics of put and send should allow the implementation the flexibility to do any of the following: 1) optimistically transmit whatever it can everytime so long as it doesn't block 2) never bother transmitting anything until you force it to by calling send 3) anything in between the first two, e.g. magically transmit once you've put enough messages to reach the optimal batch size The reason for the behaviour you are observing is that we currently do option 2 in the C impl, however we've done option 1 in the past (and I think we do option 1 still in the Java impl), and we will probably do option 3 in the future. This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith --Rafael
Re: put vs. send
On 03/05/2013 02:14 PM, Rajith Attapattu wrote: This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith Making this configurable multiplies the size of the test matrix. Can't we make this simpler? To me, this sounds like an I/O facility in which your output lines may never get sent if you don't call fflush(). This will be a surprise to most programmers, who rarely use fflush(). I think most programmers would be happier if put caused the messages to be eventually sent and send was used only for blocking until messages were flushed out. -Ted
Re: put vs. send
On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross tr...@redhat.com wrote: On 03/05/2013 02:14 PM, Rajith Attapattu wrote: This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith Making this configurable multiplies the size of the test matrix. Can't we make this simpler? I do understand your concern here, but the Java impl already does both #1 and #2 and Rafi wants to do #3 in the future. The old JMS client does something similar. I agree that if we just do option #2 (as you suggest below), then the application can easily do #1 and #3 on top of that. But I'm sure they will like if the library implements those strategies for them and they have the ability to pick a strategy. To me, this sounds like an I/O facility in which your output lines may never get sent if you don't call fflush(). This will be a surprise to most programmers, who rarely use fflush(). I think most programmers would be happier if put caused the messages to be eventually sent and send was used only for blocking until messages were flushed out. -Ted
Re: put vs. send
On Tue, Mar 5, 2013 at 11:10 AM, Ted Ross tr...@redhat.com wrote: On 03/05/2013 02:01 PM, Rafael Schloming wrote: On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish mgoul...@redhat.com wrote: So, am I understanding correctly? -- I should be able to get messages from my sender to my receiver just by calling put() -- if the receiver is ready to receive? Not necessarily, the receiver being ready just means you are unblocked on AMQP level flow control. You could also potentially block on the socket write (i.e. TCP level flow control). You need to be unblocked on both for put to succeed. Certainly there is no TCP flow control happening in Mick's scenario. What I said was put is *allowed* to send optimistically, not that it is required to. It actually did send optimistically in a previous version of the code, however I commented that line out. I would say the documented semantics of put and send should allow the implementation the flexibility to do any of the following: 1) optimistically transmit whatever it can everytime so long as it doesn't block 2) never bother transmitting anything until you force it to by calling send 3) anything in between the first two, e.g. magically transmit once you've put enough messages to reach the optimal batch size The reason for the behaviour you are observing is that we currently do option 2 in the C impl, however we've done option 1 in the past (and I think we do option 1 still in the Java impl), and we will probably do option 3 in the future. If this is the case, then Mick's original view is correct. The application must assume that messages will not ever be sent unless send is called. There is no flowing, pipelined, non-blocking producer. It's not correct as documentation of the API semantics. It's also not correct to say that messages will never be sent unless send is called, e.g. the following code will work fine: Client: m.put(request); m.recv(); // wait for reply m.get(reply); Server: while True: m.recv(); // wait for request m.get(request) m.put(reply); As for there being no flowing, pipelined, non-blocking producer, that seems like an orthogonal issue, and depending on what you mean I wouldn't say that's necessarily true either. You can certainly set the messenger's timeout to zero and then call put followed by send to get the exact same semantics you would get if put were to optimistically send every time. --Rafael
Re: put vs. send
- Original Message - On Tue, Mar 5, 2013 at 11:10 AM, Ted Ross tr...@redhat.com wrote: On 03/05/2013 02:01 PM, Rafael Schloming wrote: On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish mgoul...@redhat.com wrote: So, am I understanding correctly? -- I should be able to get messages from my sender to my receiver just by calling put() -- if the receiver is ready to receive? Not necessarily, the receiver being ready just means you are unblocked on AMQP level flow control. You could also potentially block on the socket write (i.e. TCP level flow control). You need to be unblocked on both for put to succeed. Certainly there is no TCP flow control happening in Mick's scenario. What I said was put is *allowed* to send optimistically, not that it is required to. It actually did send optimistically in a previous version of the code, however I commented that line out. I would say the documented semantics of put and send should allow the implementation the flexibility to do any of the following: 1) optimistically transmit whatever it can everytime so long as it doesn't block 2) never bother transmitting anything until you force it to by calling send 3) anything in between the first two, e.g. magically transmit once you've put enough messages to reach the optimal batch size The reason for the behaviour you are observing is that we currently do option 2 in the C impl, however we've done option 1 in the past (and I think we do option 1 still in the Java impl), and we will probably do option 3 in the future. If this is the case, then Mick's original view is correct. The application must assume that messages will not ever be sent unless send is called. There is no flowing, pipelined, non-blocking producer. It's not correct as documentation of the API semantics. It's also not correct to say that messages will never be sent unless send is called, e.g. the following code will work fine: Client: m.put(request); m.recv(); // wait for reply m.get(reply); That recv() just caused an enqueued message to be sent? I just tried it in a C program and it worked, more or less. At least the receiver got his 5 messages without send() ever having been called. Does a call to recv() also cause all unblocked messages to be sent? Is that symmetric? Does a call to send() also cause messages to be received ? Server: while True: m.recv(); // wait for request m.get(request) m.put(reply); As for there being no flowing, pipelined, non-blocking producer, that seems like an orthogonal issue, and depending on what you mean I wouldn't say that's necessarily true either. You can certainly set the messenger's timeout to zero and then call put followed by send to get the exact same semantics you would get if put were to optimistically send every time. --Rafael
Re: put vs. send
On Tue, Mar 5, 2013 at 3:20 PM, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu rajit...@gmail.comwrote: On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross tr...@redhat.com wrote: On 03/05/2013 02:14 PM, Rajith Attapattu wrote: This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith Making this configurable multiplies the size of the test matrix. Can't we make this simpler? I do understand your concern here, but the Java impl already does both #1 and #2 and Rafi wants to do #3 in the future. The old JMS client does something similar. I agree that if we just do option #2 (as you suggest below), then the application can easily do #1 and #3 on top of that. But I'm sure they will like if the library implements those strategies for them and they have the ability to pick a strategy. I don't see why we'd make this configurable. All three options actually fit the same general semantics. Even if you're optimistically trying to transmit every single time put is called it's entirely possible for the socket to be blocked every single time you try. If this were to happen the implementation of #1 would appear to behave precisely the same as #2 behaves. In other words if you're coding correctly against the API you can't assume that put will or won't have transmitted anything regardless of which strategy is used internally. I agree with you. You make a very good point. Perhaps we should explicitly make that clear in our docs to avoid applications written against wrong assumptions. Rajith --Rafael
Re: put vs. send
- Original Message - On Tue, Mar 5, 2013 at 3:20 PM, Rafael Schloming r...@alum.mit.edu wrote: On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu rajit...@gmail.comwrote: On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross tr...@redhat.com wrote: On 03/05/2013 02:14 PM, Rajith Attapattu wrote: This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith Making this configurable multiplies the size of the test matrix. Can't we make this simpler? I do understand your concern here, but the Java impl already does both #1 and #2 and Rafi wants to do #3 in the future. The old JMS client does something similar. I agree that if we just do option #2 (as you suggest below), then the application can easily do #1 and #3 on top of that. But I'm sure they will like if the library implements those strategies for them and they have the ability to pick a strategy. I don't see why we'd make this configurable. All three options actually fit the same general semantics. Even if you're optimistically trying to transmit every single time put is called it's entirely possible for the socket to be blocked every single time you try. If this were to happen the implementation of #1 would appear to behave precisely the same as #2 behaves. In other words if you're coding correctly against the API you can't assume that put will or won't have transmitted anything regardless of which strategy is used internally. I agree with you. You make a very good point. Perhaps we should explicitly make that clear in our docs to avoid applications written against wrong assumptions. I can certainly do that, but it seems to me that semantics should be simple, obvious, and orthogonal. What seems non-simple and non-obvious to me so far is: put() might send, or not. It doesn't send now, but it might later. recv() can cause messages to be sent. send() can cause messages to be received. I would think that 1. every verb should only mean one thing 2. there should be a simple mental model, against which every verb performs a predictable action. so for example: put( messenger, message ); // enqueue for sending send ( messenger, BLOCK ); // block till all sent. send ( messenger, DONT_BLOCK ); // send what you can. credit ( messenger, 10 ); // limit incoming queue size recv ( messenger, BLOCK ); // block till I get a message recv ( messenger, DONT_BLOCK ); // if no messages incoming, return. Rajith --Rafael
Re: put vs. send
On Tue, Mar 5, 2013 at 12:39 PM, Michael Goulish mgoul...@redhat.comwrote: - Original Message - On Tue, Mar 5, 2013 at 11:10 AM, Ted Ross tr...@redhat.com wrote: On 03/05/2013 02:01 PM, Rafael Schloming wrote: On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish mgoul...@redhat.com wrote: So, am I understanding correctly? -- I should be able to get messages from my sender to my receiver just by calling put() -- if the receiver is ready to receive? Not necessarily, the receiver being ready just means you are unblocked on AMQP level flow control. You could also potentially block on the socket write (i.e. TCP level flow control). You need to be unblocked on both for put to succeed. Certainly there is no TCP flow control happening in Mick's scenario. What I said was put is *allowed* to send optimistically, not that it is required to. It actually did send optimistically in a previous version of the code, however I commented that line out. I would say the documented semantics of put and send should allow the implementation the flexibility to do any of the following: 1) optimistically transmit whatever it can everytime so long as it doesn't block 2) never bother transmitting anything until you force it to by calling send 3) anything in between the first two, e.g. magically transmit once you've put enough messages to reach the optimal batch size The reason for the behaviour you are observing is that we currently do option 2 in the C impl, however we've done option 1 in the past (and I think we do option 1 still in the Java impl), and we will probably do option 3 in the future. If this is the case, then Mick's original view is correct. The application must assume that messages will not ever be sent unless send is called. There is no flowing, pipelined, non-blocking producer. It's not correct as documentation of the API semantics. It's also not correct to say that messages will never be sent unless send is called, e.g. the following code will work fine: Client: m.put(request); m.recv(); // wait for reply m.get(reply); That recv() just caused an enqueued message to be sent? I just tried it in a C program and it worked, more or less. At least the receiver got his 5 messages without send() ever having been called. Does a call to recv() also cause all unblocked messages to be sent? When a messenger is blocked it will do whatever outstanding work it can. This won't necessarily cause all messages to be sent, but it may cause some. Is that symmetric? Does a call to send() also cause messages to be received ? Yes it is symmetric, messages can arrive while you're blocking for send(). --Rafael
Fwd: put vs. send
Oops, meant to send this to the list.
-- Forwarded message --
From: Rafael Schloming r...@alum.mit.edu
Date: Tue, Mar 5, 2013 at 6:44 PM
Subject: Re: put vs. send
To: Bozo Dragojevic bo...@digiverse.si
On Tue, Mar 5, 2013 at 3:25 PM, Bozo Dragojevic bo...@digiverse.si wrote:
Wow, by not ever calling pn_messenger_send(), but only pn_messenger_recv()
things are, unexpectedly, better! I'll explain below what 'better' means.
But first, this begs the question, what is the purpose of
pn_messenger_send()
and where (and why) it's appropriate/required to call it.
Well one example would be if you just want to send a bunch of messages
without ever receiving any.
I think it helps if you conceptually split the API along
blocking/non-blocking lines:
non-blocking: put, subscribe, get, incoming, outgoing, etc...
blocking: send, recv
Of the non blocking portion there are things that simply access data, like
get(), incoming, outgoing, etc, and then there are things that
schedule/request asynchronous work to be done, namely put() and subscribe().
For the blocking portion, I think it helps to understand that there is
really only one fundamental operation, and that is
do_any_outstanding_work_until_condition_X_is_met. Currently the blocking
portion of the API exposes exactly two conditions, namely all messages are
sent via send() and there is a message waiting to be received via recv().
So in your example below if you're basically just a request/response server
then as you've noticed you never actually need to call send since you don't
really care about waiting for that condition, and any work you schedule in
your handlers will be performed anyways as you wait for messages to be
received.
The results are for slightly modified 0.3 release.
Most notably, I have a local change that exposes pn_driver_wakeup()
through the messenger api.
I'm curious about why you had to make this change. Can you describe the
usage/scenario that requires it?
Our API is threaded internally but all proton communication is done
via a dedicated thread that runs the following (stub) event loop:
(The same event loop is used by both, the client library, and the 'broker')
while(1) {
ret = pn_messenger_recv(m,100) // the 100 is hard to explain...
if (ret != PN_WAKED_UP) {// new error code for wakeup case
/*
* apparently there's no need to call send...
* pn_messenger_send(m);
*/
}
Command cmd = cmd_queue.get(); // cmd_queue.put() from another thread
// will call pn_driver_wakeup() and
will
// break out of pn_messenger_recv()
if (cmd)
handle(cmd); // ends up calling pn_messenger_put()
if (pn_messenger_incoming(m)) {
msg = pn_messenger_get(m);// handle just one message
// pn_messenger_recv() will not block
// until we're done
handle(msg); // can end up calling
pn_messenger_put()
}
}
So, before the change, a test client that produced messages needed to
throttle a bit, about 8ms between each 'command' that resulted in
one 'pn_messenger_put()'
If a lower delay (or no delay) was used, the client's messenger got
confused
after some fairly small number of messages sent (order of magnitude 10)
and ended up sitting in pn_driver_wait while it had unsent messages to
send.
This sounds like a bug to me. As I mentioned above, you shouldn't need to
call send, but it also shouldn't hurt.
With the one line change of commenting out the send() it can go full speed!
I know it's hard to comment on out-of-tree modified pseudo code, but
is such an event loop within the design goals of the messenger?
Yes, certainly. Your code looks to me like it is almost the same as the
server.py example.
Longer term we'll most likely be switching from messenger to engine + driver
so we can go multithreaded with the event loop.
You may find there is a fair amount you would need to reimplement if you
were to make that switch. One of the features I'd like to get into
messenger soon is factoring the messenger implementation and adding to the
API so that you can supply your own driver and write your own event loop
based on the messenger API rather than having to go all the way down to the
engine API. If what you want is more control over the I/O and threading but
you're happy with the protocol interface provided by messenger, then this
might be a better option for you than using the engine directly.
--Rafael
