On 3/6/13 3:45 AM, Rafael Schloming wrote:
Oops, meant to send this to the list.
---------- Forwarded message ----------
From: Rafael Schloming<r...@alum.mit.edu>
Date: Tue, Mar 5, 2013 at 6:44 PM
Subject: Re: put vs. send
To: Bozo Dragojevic<bo...@digiverse.si>
On Tue, Mar 5, 2013 at 3:25 PM, Bozo Dragojevic<bo...@digiverse.si> wrote:
Wow, by not ever calling pn_messenger_send(), but only pn_messenger_recv()
things are, unexpectedly, better! I'll explain below what 'better' means.
But first, this begs the question, what is the purpose of
pn_messenger_send()
and where (and why) it's appropriate/required to call it.
Well one example would be if you just want to send a bunch of messages
without ever receiving any.
I think it helps if you conceptually split the API along
blocking/non-blocking lines:
non-blocking: put, subscribe, get, incoming, outgoing, etc...
blocking: send, recv
Of the non blocking portion there are things that simply access data, like
get(), incoming, outgoing, etc, and then there are things that
schedule/request asynchronous work to be done, namely put() and subscribe().
For the blocking portion, I think it helps to understand that there is
really only one fundamental operation, and that is
do_any_outstanding_work_until_condition_X_is_met. Currently the blocking
portion of the API exposes exactly two conditions, namely "all messages are
sent" via send() and "there is a message waiting to be received" via recv().
So in your example below if you're basically just a request/response server
then as you've noticed you never actually need to call send since you don't
really care about waiting for that condition, and any work you schedule in
your handlers will be performed anyways as you wait for messages to be
received.
Thanks for this really good explanation. This plus your proposed change in
the other part of the thread would make the API much more
The results are for slightly modified 0.3 release.
Most notably, I have a local change that exposes pn_driver_wakeup()
through the messenger api.
I'm curious about why you had to make this change. Can you describe the
usage/scenario that requires it?
Our design has a 'protocol' thread that is separate from the API,
the idea being we can already be processing next incoming network
messages while the API user is still busy servicing the previous one.
By processing I mean all the CPU-required deserialization work of
converting the network packets first to pn_message() and then to our
internal higher-level representation of the message content.
The mechanism how API thread sends a message is to hand off such a
high-level objects off to the 'proto' thread via a queue.
so we end up with a (proto) thread that has two wakeup sources
a) network related events (ability to send more stuff or receive it), and
b) commands from the API coming from the queue.
Idle state for the proto thread is no network activity and no API
activity and in that state we want to be blocked somewhere. We choose
pn_driver_wait() as it's a natural wait point and also we can have best
possible incoming message latency.
We also need to react quickly to messages that API wants to send out.
Given the messenger is hiding it's driver it seemed a smaller change
to expose the pn_driver_wakeup() via messenger (and teach messenger
to actually come out of it's tsync method).
Even if we did add locking and would be able to call pn_messenger_put()
from a separate thread, while we left the 'proto' thread blocked in
pn_driver_wait() it wouldn't help as the poll mask would be wrong --
if we were not sending before then pn_driver_rebuild() had not selected
writable condition for that connection -- or we wouldn't be blocked in
pn_driver_wait() -- talking about a mostly-idle low latency situation
where TCP buffers are not clogged up.
hope this clarifies things more than it muddles them up :)
Our API is threaded internally but all proton communication is done
via a dedicated thread that runs the following (stub) event loop:
(The same event loop is used by both, the client library, and the 'broker')
while(1) {
ret = pn_messenger_recv(m,100) // the 100 is hard to explain...
if (ret != PN_WAKED_UP) { // new error code for wakeup case
/*
* apparently there's no need to call send...
* pn_messenger_send(m);
*/
}
Command cmd = cmd_queue.get(); // cmd_queue.put() from another thread
// will call pn_driver_wakeup() and
will
// break out of pn_messenger_recv()
if (cmd)
handle(cmd); // ends up calling pn_messenger_put()
if (pn_messenger_incoming(m)) {
msg = pn_messenger_get(m); // handle just one message
// pn_messenger_recv() will not block
// until we're done
handle(msg); // can end up calling
pn_messenger_put()
}
}
So, before the change, a test client that produced messages needed to
throttle a bit, about 8ms between each 'command' that resulted in
one 'pn_messenger_put()'
If a lower delay (or no delay) was used, the client's messenger got
confused
after some fairly small number of messages sent (order of magnitude 10)
and ended up sitting in pn_driver_wait while it had unsent messages to
send.
This sounds like a bug to me. As I mentioned above, you shouldn't need to
call send, but it also shouldn't hurt.
Agree, it is a bug, but right now sidestepping it was my favorite option :)
With the one line change of commenting out the send() it can go full speed!
Replying to myself, I need to explain that implication of my conculusion was
a bit flawed. The change allows the *producer* to go full speed.
In reality this means that we're calling pn_messenger_put() at a much higher
rate than what network can possibly handle. So the short term effect is
producer grows out of band and doesnt really do anything good to the latency
of sent messages.
So for me this opens up two new areas (one of the reasons it took me a
bit to reply):
- study up the API on 'trackers',
iiuc thats the name of what pn_messenger_put() returns
- figure out how to use them inside our code so that
the publisher (API user) will not drown itself in messages
it cannot realistically send.
I know it's hard to comment on out-of-tree modified pseudo code, but
is such an event loop within the design goals of the messenger?
Yes, certainly. Your code looks to me like it is almost the same as the
server.py example.
yay :)
Longer term we'll most likely be switching from messenger to engine + driver
so we can go multithreaded with the event loop.
You may find there is a fair amount you would need to reimplement if you
were to make that switch.
I know, it's 1500 lines of not totally straightforward code that at this
moment I'm glad I don't need to write :)
One of the features I'd like to get into
messenger soon is factoring the messenger implementation and adding to the
API so that you can supply your own driver and write your own event loop
based on the messenger API rather than having to go all the way down to the
engine API. If what you want is more control over the I/O and threading but
you're happy with the protocol interface provided by messenger, then this
might be a better option for you than using the engine directly.
there's a few separate things, with overall goal being low latency e2e.
- ability to spread out load of serialization and deserialization over cores
and as mentioned above there are two 'layers' of it:
bytes <=> pn_message <=> API object
Providing this within the messenger paradigm sounds tricky to me
-- I definitely don't want messenger to grow it's own thread pool :)
- more control over IO and waiting (as per description above)
this alone would still fit into your proposed extension quite nicely
In the other thread you mention bringing send() and recv() closer wrt
meaning of
a parameter.
It'd be a tiny step from there to provide a sync(send_limit, recv_limit)
which would help me in reasoning of there being exactly one path that
ends up
in pn_driver_wait() -- this would simplify interactions between
- ability to maybe (ab)use several TCP connections for sending of messages
with different latency-sensitive information
This one is more a possible solution than a requirement
What would be sufficient from interface perspective is to have
ability for
messages (potentially to the same destination) to jump the queue, so
to speak,
but this could potentially still fit within the messenger paradigm
Thanks for all the insights,
Bozzo
