Hi,
I'll do my best to answer what I can... comments inline...
On Fri, Oct 31, 2014 at 8:10 AM, Fraser Adams <[email protected]
wrote:
Hey all,
OK I'll 'fess up I have to admit that although I've been tinkering with
Messenger for a while now I don't *really* understand some of the terms
that get used such as credit, disposition, settlement. I think that I was
OK with qpid::messaging's setCapacity stuff and how to use that to optimise
prefetch and also using qpid::messaging link controls in the Address string
to set/disable reliability but the settings on Messenger remain a mystery
to me.
What really brought that home in my mind was when I started playing with
the msgr-send and msgr-recv applications in <proton>/tests/tools/apps to
try to figure what sort of throughput I might get with Messenger.
First off let me just say that I'm not that familiar with msgr-send and
msgr-recv. I think Ken wrote those, and as I understand it they are command
line sending and receiving programs that happen to be implemented via
messenger, however they aren't intended to be command line interfaces to
messenger, so the terminology and parameters, may not line up 100% with the
messenger api-doc.
A while back Gordon was involved in a performance conversation and
mentioned testing with the settings below
./msgr-recv -c 1000000
./msgr-send -c 1000000 -b 64
So sending/receiving a million 64 octet messages and seeing what the
performance is - so far so good.
But I then tinkered around and hacked in some code to display the count
for the sent and received messages and then did:
./msgr-recv -c 100
./msgr-send -c 100 -b 1000000
in other words sending & receiving 100 1MB messages - I actually used the
large message size as much to slow things down as anything, but what I
observed was that the 100 messages were all being sent before msgr-recv
started to display any received count numbers.
When I looked at the usage I noticed the -p option "Send batches of #
messages" and sure enough if I do
./msgr-send -c 100 -p 10 -b 1000000
I see msgr-recv catch up every 10 messages.
What I *think* is going on is that when the count in the internal
Messenger queue (pn_messenger_outgoing(messenger)) exceeds the batch size
it calls pn_messenger_send(messenger, -1);
Yes, from my brief perusal of the code I would concur.
But that makes me unclear in my mind what the differences between
pn_messenger_put and pn_messenger_send actually are, I've certainly seen
pn_messenger_put actually send messages. I realise that there's a comment
"The message may also be sent if transmission would not cause blocking" but
I'm not clear at exactly which point blocking would occur, I'm guessing
that I'm noticing this because of my large messages? The problem of course
is that if I use tiny little messages I can't actually see if any batching
actually occurs or whether in the small message case pn_messenger_put
merrily whizzes out the small messages without really needing
pn_messenger_send to give them a helping kick.
I'm a little confused by this question. The pn_messenger_put operation
hands a new message over to the messenger and as stated in the API doc is
guaranteed not to block. The pn_messenger_send operation on the other hand
does not take a new message, and exists solely for the purpose of blocking
until previously "put" messages are actually sent. The pn_messenger_send
operation is the equivalent of
pn_mesenger_work_until_you_send_N_messages(N).
Does that make sense? It'd be useful for someone who knows this stuff to
explain how the Messenger store works and how the various API calls relate
to credit, disposition & settlement (I'm pretty sure the latter relates to
the tracker/window/accept/settle stuff but not so sure about the first
two). I'd also quite like to know how this stuff relates to the
capacity/reliability stuff on qpid::messaging.
Credit refers to credit based flow control. In a credit based flow control
scheme, the receiver maintains a "credit balance". The credit balance is
just a number that indicates how many messages the receiver is capable of
receiving at any given point in time. The receiver periodically informs the
sender of this number, and the sender guarantees never to send unless this
number is positive. The sender will also decrement its copy of the credit
balance whenever it sends a message. This guarantees that the sender will
never send more messages than the receiver has requested.
So the term credit can mean a bunch of different things in different
contexts, but when used as a "unit", e.g. 10 credits, a credit pretty much
translates into permission to send one message. So when the receiver issues
10 credits, it is issuing permission to send up to 10 messages.
Now depending on what exact policy you use to issue credits, you can
implement a lot of different semantics, e.g. you can send 1 credit to fetch
exactly one message and be done with it, or you can renew the sender's
credit whenever it falls below a certain threshold, and there are many more
options. The concept of capacity is just a specific policy for managing
credits. This policy assumes the receiver has a fixed buffer for holding
messages, and it issues credit to reflect the number of empty slots in that
buffer.
Settlement of a delivery refers to an endpoint being done with and
forgetting everything about a given delivery. Disposition refers to the
state of the delivery at the time of settlement, e.g. was it accepted vs
rejected. Sometimes the disposition is null because the state at the time
of settlement is unknown, e.g. the sender can choose to "pre-settle" the
message as it is sent, i.e. forget about it as soon as it hits the wire. In
this case the state when the receiver settles it will never be known. Just
like credit is a lower level/more general concept than capacity and can be
used to implement a greater variety of semantics, settlement is a lower
level/more general concept than reliability and can be used to implement a
variety of different QoS levels.
Also I *think* that there is a problem with the python version of
msgr-send.py I'd expect that to run more slowly than the C version, but
when I did:
./msgr-recv.py -c 100
./msgr-send.py -c 100 -b 1000000
it returned more or less immediately and when I increase the -c value I
appear to be seeing the same throughput irrespective of the value of the -b
value. I've not really looked too deeply at the code but I wonder if that
rings any bells for anyone?
Doesn't ring a bell for me. I'd file a JIRA and maybe poke Ken.
Sorry if these things are obvious to the people who know, but I figured I
probably wasn't the only one who didn't actually know this stuff and as
I've got no shame I thought I'd raise my head above the parapet and expose
my ignorance to the world :-)
I hope this helps a bit, please follow up and let me know if you have
further questions.
--Rafael
