On 11/01/13 22:36, Lance D. wrote:
I love this discussion. It's actually very helpful in getting a better
grasp of where QPID fits into the bigger picture of the AMQP world.
I'm going to inject a little more philosophical question into the nuance
here. I completely get what Andy said about the standard and it makes
sense. However, what frustrates me is that, as the API is implemented in
QPID, internally QPID has access to exactly what I want, it was just
overlooked or ignored when it comes to storing that detail.
So I think that you have to consider the difference between the API and
what's actually going on on the wire and in the broker. That's
specifically why I said in my earlier post that _the API_ can lead to
the sort of confusion that we've both suffered.
The key thing is that if you want to *really* get under the skin of
what's going on in your _system_ you have to consider the client
application, the client runtime (e.g. the API implementation such as
qpid::messaging or the JMS Impl) and the broker. If you only consider
one of these you end up making assumptions which, when things end up not
behaving as "expected", end up making diagnosing the problem a whole lot
harder.
So in this case you need to realise that "the API is implemented in
QPID" is really the client runtime environment that I refer to above.
That API results in AMQP frames being sent over the wire. So whilst it's
true to say that the client runtime has access to the information that
we might want it's *not* true to extrapolate that to assuming that the
broker has access to the same information. The broker and the client
runtime talk to each other purely in AMQP terms which is why Andy is
entirely correct to be talking in AMQP terms in terms of what
information the QMF management objects maintained on the broker actually
have access to. So I don't think your assertion "it was just overlooked
or ignored when it comes to storing that detail." holds at all here.
As an aside another reason why I say it really helps to consider the
system of systems relates to a scenario that I had when trying to
understand why one of my brokers appeared to have a "memory leak". I'd
employed consumer pre-fetching (which is what most people do for
efficiency). What bit me was that it's the client *application* that
does the acknowledgements, so when I had a slow consumer my broker used
way more memory than I thought it should because I was using ring
queues. What was actually going on though was that the broker also has
to maintain references to messages that have been prefetched but not yet
acknowledged (so it can resend them if the consumer fails). If you have
lots of consumers that are slow relative to producers this can use way
more memory than you "expect".
So it's a bit of a rambly aside but hopefully it illustrates why it's a
good idea to consider the whole system. That scenario took me an
absolute age to figure out so hopefully that little bit of advice gives
you a leg up when you need to diagnose your own issues.
I say that
because my tools can capture the connection events and collect enough
detail that later I can see what publisher sent the message. However, in
order for that to work, my tools must be running before anyone connects;
that is less useful for use debugging a large system of systems.
So my connection audit has a similar need to be running before anyone
connects, the obvious approach is to start up qpidd from a script which
also then fires up your QMF client.
Aside from the standard's details that allow a connection to send to any
queue,
A producer connection sends to exchanges not queues. Even in the case of
the default direct exchange where the producer address "looks like" a
queue name what's actually happening is that the message is getting sent
to default direct exchange with the queue name as the binding key.
That's another case of the APIs abstracting some of the detail away.
is there are reason that QPID doesn't keep some sort of mapping that
maintains what was declared at connection time and/or the most recent
destination?
As I said at the start of this mail that relates to the difference
between the client run time and the broker, so the client runtime I
suspect does "keep some sort of mapping that maintains what was declared
at connection " 'cause when it does sender.send() it needs to find the
session and connection associated with the sender, but once again the
association that is formed in the client runtime translates to a
*transient* association when the message is sent over the wire and
that's all the broker has to work with - and it's the broker that
maintains the QMF management objects.
So Qpid _as a system_ does conceptually have that mapping 'cause that's
how the API works, but the Qpid broker does not, if you see what I mean?
Re "the most recent destination " I guess that what you really mean here
is could the session object maintain some sort of exchangeRef in a
similar way to the way subscription maintains a queueRef? It's an
interesting thought because I suspect most use-cases are likely to
involve producer connections publishing to a single exchange (or at
least producer sessions doing that) so the "most recent association"
might be useful. However I suspect it's an idea unlikely to gain any
traction because it's pretending that what is really a transient
association is really a full association (think in UML terms here) so
the modeling is not accurate, moreover whilst in most cases the
multiplicity between session and exchange is likely to be one it's
really a zero to many transient association.
Thanks and happy weekend folks!
-Lance
On Fri, Jan 11, 2013 at 12:39 PM, Fraser Adams <
fraser.ad...@blueyonder.co.uk> wrote:
Yes, with AMQP 0-10, the only time a producer indicates the destination of
the message is when it actually sends the message. The message.transfer
command includes the destination of the message, which is an exchange. A
producer may send to various exchanges, simply by changing the destination
in the message.transfer command with each message. As a result, from the
broker's perspective, there's not really any way to look at a producer
client and determine which exchange(s) it's sending to.
Thanks Andy that kind of makes sense.
I think that part of the confusion might stem from some of the higher
level APIs, I've only really used JMS and qpid::messaging (I've steered
well clear of qpid::client as recommended by Gordon and several others and
I know plenty of people who've been bitten by qpid::client which makes me
glad I did :-)).
So for qpid::messaging and JMS the model tends to be that one creates a
connection, then one creates a session and in general in qpid::messaging
one would create Senders specifying an address and in JMS similarly one
would create a MessageProducer. So I guess that it's fairly easy to
*assume* that the address (which in the case of a producer could just be
the exchange name for something like amq.match) is associated with the
Connection/Session.
if you normally do:
connection.open();
Session session = connection.createSession();
Sender sender = session.createSender("amq.**match");
........
sender.send(message);
.........
it's easy to fall into that trap.
I guess that the alternate send which actually takes an address (e.g. the
JMS MessageProducer |*send <http://docs.oracle.com/**
javaee/5/api/javax/jms/**MessageProducer.html#send%**
28javax.jms.Destination,%**20javax.jms.Message%29<http://docs.oracle.com/javaee/5/api/javax/jms/MessageProducer.html#send%28javax.jms.Destination,%20javax.jms.Message%29>
*(**Destination <http://docs.oracle.com/**javaee/5/api/javax/jms/**
Destination.html<http://docs.oracle.com/javaee/5/api/javax/jms/Destination.html>>
destination, Message <http://docs.oracle.com/**javaee/5/api/javax/jms/**
Message.html <http://docs.oracle.com/javaee/5/api/javax/jms/Message.html>>
message)| ) is a closer analogue to the underlying AMQP message.transfer I
guess that the higher level abstraction just acts as a cache for the real
AMQP destinations.
Thanks again Andy I think that your explanation has helped clarify in my
mind the transient association between Connection and Exchange even though
I suspect in practice the vast majority of producer code is likely to be
written in a way that retains what looks like a more sustained association,
but that's only in the client runtime not in the broker.
Frase.
