On 3/8/13 3:19 PM, Rafael Schloming wrote:
On Thu, Mar 7, 2013 at 5:15 AM, Bozo Dragojevic <bo...@digiverse.si
<mailto:bo...@digiverse.si>> wrote:
On 3/6/13 3:45 AM, Rafael Schloming wrote:
Oops, meant to send this to the list.
---------- Forwarded message ----------
From: Rafael Schloming<r...@alum.mit.edu <mailto:r...@alum.mit.edu>>
Date: Tue, Mar 5, 2013 at 6:44 PM
Subject: Re: put vs. send
To: Bozo Dragojevic<bo...@digiverse.si
<mailto:bo...@digiverse.si>>
[snip]
Thank you for taking the time to explain, that's very helpful. It
makes me wonder if we shouldn't include some kind of
pn_messenger_interrupt() that to pull out of blocking calls early.
That'd be great!
Few weeks ago I've submitted
https://issues.apache.org/jira/browse/PROTON-231
for this feature and the JIRA has a link to my attempt at implementation
(based off 0.3 branch) -- this is what we use now.
I've named it pn_messenger_wakeup() after pn_driver_wakeup() that it calls.
In addition the pn_messenger_tsync() needed to be adjusted to actually
exit early :)
So for me this opens up two new areas (one of the reasons it took
me a bit to reply):
- study up the API on 'trackers',
iiuc thats the name of what pn_messenger_put() returns
- figure out how to use them inside our code so that
the publisher (API user) will not drown itself in messages
it cannot realistically send.
I'm not sure trackers are what you want here (although it may be
depending on your app). Trackers are used to track the status of the
message at the receiver, e.g. it will tell you whether the receiver
has accepted or rejected the message.
the fourth paragraph below circles back to trackers :)
If you just care about limiting the rate of your producer to match
what you can actually push onto the wire, then you can just look at
pn_messenger_outgoing(), that will tell you how many messages are
actually backed up waiting to go onto the wire, e.g. before or after
doing a put/send/recv/etc, you can check the size of the outgoing
queue and based on that decide whether you need to throttle your producer.
Looking at total size of outgoing queue is sufficient for the messenger
that is
part of the application that acts as the publisher and sends messages to the
broker as it then has just one peer.
The scenario that is important is this: Broker, too, wants to rate-limit
each peer
based solely on that peer's capability to receive messages. So if there
are two
subscribers and one is on a slower link, or is not behaving in some
other way,
then the fast subscriber should not be held hostage of the slow subscriber.
This is even more so if we're talking about two publishers each having a
set of own subscribers.
On the messenger level, the broker ends up having a bunch of amqp
addresses where
the message received from the publisher need to be forwarded.
The answer that is sought is pn_messenger_outgoing(address). Now, a
tracker does
uniquely identify the remote address that is embedded in the message
being sent.
If there is/were a query of local messenger's opinion on progress status
wrt.
delivering of that one message where one distinct answer would be
"managed to put it on the wire".
A function like that makes it trivial for a messenger user (broker) to
perform
the accounting of 'number of outstanding messages per address' on it's own.
- ability to spread out load of serialization and deserialization
over cores
and as mentioned above there are two 'layers' of it:
bytes <=> pn_message <=> API object
Providing this within the messenger paradigm sounds tricky to me
-- I definitely don't want messenger to grow it's own thread
pool :)
This sounds to me like it could possibly be addressed by a smarter
pn_message implementation. Right now pn_message_t is implemented in a
very basic way where it actually does a lot of unnecessary
encode/decode work. Not only could that be made more efficient so
there is much less load to be distributed, but it could also be made
to happen on demand. That would give you the flexibility of hitting
most of the cost of the bytes <=> pn_message_t portion in whichever
thread you chose.
This multi-core business obviously makes most sense for cases where
there is more
than one TCP connection where data flows in.
The thread that read a bunch of bytes from a socket already has them in
that CPU cache.
Doing as much of the work with those bytes on that CPU, without context
switches,
will be fast.
- more control over IO and waiting (as per description above)
this alone would still fit into your proposed extension quite nicely
In the other thread you mention bringing send() and recv()
closer wrt meaning of
a parameter.
It'd be a tiny step from there to provide a sync(send_limit,
recv_limit)
which would help me in reasoning of there being exactly one path
that ends up
in pn_driver_wait() -- this would simplify interactions between
- ability to maybe (ab)use several TCP connections for sending of
messages
with different latency-sensitive information
This one is more a possible solution than a requirement
What would be sufficient from interface perspective is to have
ability for
messages (potentially to the same destination) to jump the
queue, so to speak,
but this could potentially still fit within the messenger paradigm
This is an interesting one AMQP does provide a priority header on the
message that is a clue that ordering is relaxed. Right now neither
messenger nor the engine pays any attention to it though. As an
interface it seems like the obvious thing, however implementation of
it might be interesting.
Thanks,
Bozzo
