On Jan 11, 2013, at 10:15 AM, Fraser Adams wrote: > On 11/01/13 13:11, Lance D. wrote: >> Wow. Thanks for the huge amount of detail! > No worries, hope it helps. One of the best things about Qpid is the strong > user community, I've got a lot out of it so it's nice to be able to give a > little back. >> >> You are correct: I am using the C++ broker. As far as language for my >> tool, I'm pretty agnostic; I want a tool that suits my needs and I've got >> the resources to do it in any viable language (though I am partial to C++). > Unfortunately C++ has the poorest QMF support IMHO :-( There is an API, but > it bears no resemblance to the QMF2 API that I linked previously :-( I've not > done much with that I'm afraid. I keep promising myself that I'll port my > Java implementation to C++, but until I get more feedback on my Java stuff > it's hard to find the motivation..... > > The is an example using the C++ QMF API in > qpid/cpp/bindings/qmf2/examples/cpp but as I say I'm not overly familiar with > that API and haven't used it myself. > > I've attached a few examples that Gordon Sim put together that use the QMF2 > protocol directly from C++ which you might find useful. > > Java and python currently have the best QMF support, though as I said > previously I'll shortly be releasing a REST API for QMF2 and also JavaScript > support :-) > >> >> I'm going to follow up on one paragraph in particular: >> "To more directly answer your question "However, I’m having a hard time >> making that same mapping between the connection stuff and a sender to an >> exchange. " yeah there's some nuance there :-) the main thing is to think >> in terms of UML associations, so for example a binding object has queueRef >> and exchangeRef properties which hold ObjectIds that enable the associated >> queue and exchange to be indexed (binding behaves like a UML association >> class really) what this means though if you want to work out how a queue >> links to an exchange you have to do it from the perspective of the binding." >> >> My problem is that my publishers establish a 'send only' connection, so >> they link to the exchange directly (without a queue). To put it another >> way, if I have n publishers sending to exchange 'myExchange', when I run >> 'qpid-stat -q' I don't see n queues bound to myExchange. So, without a >> queueRef, I have no binding object for those publishers, and no way to get, >> for example, the remote PID. Am I missing something? > Ahhh so what I think you're saying is that you want to figure out linkage > from producer connections. > > Unfortunately I don't think that you're missing something. So as I said > previously it's possible (though not trivial) to work out associations > between consumer connections and the queues that they are getting their data > from, but there isn't any good way to do what might reasonably be considered > the converse which is to work out the exchange that a producer connection is > publishing to. > > I asked about that *ages* ago - I might be able to find the post in my sent > box but it'll probably take a while. IIRC the response that I got from Gordon > Sim was that it related to the AMQP spec and that there wasn't a mechanism to > do this.
Yes, with AMQP 0-10, the only time a producer indicates the destination of the message is when it actually sends the message. The message.transfer command includes the destination of the message, which is an exchange. A producer may send to various exchanges, simply by changing the destination in the message.transfer command with each message. As a result, from the broker's perspective, there's not really any way to look at a producer client and determine which exchange(s) it's sending to. > > So basically I don't think that there's any way to look up an exchange > associated with a connection. > > The closest that I've got to that is to at least flag up that a connection > relates to a "producer only" client by checking for connections that have > sessions but no subscribers. > > The remotePid is a property of the connection management object. > > if you only have producer clients (isn't that a bit useless with nothing > consuming the messages??) all you will see in that part of the graph is > connections and sessions. > > > I wish that I could give a happier answer, but I'm close to certain that you > aren't missing something. I've had users ask for this sort of thing myself > and have had to say it's not possible, I've not even worked out a reliable > way to even *infer* the exchange. As I say the closest I got was to at least > work out which of my connections only related to producers so I could work > out the remote host and pid of those. > > Perhaps Gordon or one of the others on the core Qpid team has some more > insight, but I suspect I've probably done more with QMF than most. > > Sorry I can't give the answer you'd like, > Frase > >> >> Thanks again, >> -Lance >> >> On Fri, Jan 11, 2013 at 5:44 AM, Fraser Adams <[email protected] >>> wrote: >>> Howdy Lance, >>> >>> You mention qpid-tool which uses QMF so I'm assuming that you are talking >>> about the C++ broker. The Java broker uses JMX and I'm not so familiar with >>> how that does instrumentation so the following is only really applicable to >>> the C++ broker. >>> >>> >>> Unfortunately QMF can be a bit non-trivial, it's not helped by the fact >>> that there are two variants - QMF1 which is a binary protocol and >>> accompanying API and QMF2 which is built on top of Map messages. >>> >>> The python tools bundled with qpid used to use QMF1, though I *think* that >>> as of qpid 0.18 they've started to use QMF2 (trying 0.18 qpid-config on a >>> 0.8 broker gets a method failed error which suggests it's using QMF2 create >>> method - though I've not actually looked at the 0.18 source). >>> >>> If you are writing your tools in python there's a whole lot more stuff >>> available for QMF. >>> >>> If you are writing in Java a year back I wrote a complete Java >>> implementation of the QMF2 API: >>> https://cwiki.apache.org/qpid/**qmfv2-api-proposal.html<https://cwiki.apache.org/qpid/qmfv2-api-proposal.html> >>> >>> based on the protocol: >>> https://cwiki.apache.org/qpid/**qmf-map-message-protocol.html<https://cwiki.apache.org/qpid/qmf-map-message-protocol.html> >>> >>> The link to my code is here: >>> https://issues.apache.org/**jira/browse/QPID-3675<https://issues.apache.org/jira/browse/QPID-3675> >>> >>> >>> Slightly frustratingly there hasn't been much feedback on this - I'd >>> really like to see it make it into the main code base - I put a huge amount >>> of effort into it. The code in that Jira link also contains tests/tools >>> that illustrate how to use QMF2 in Java in some non-trivial scenarios. >>> >>> >>> As it happens in a few days time I'm about to release a major uplift to >>> that, which will include a QMF2 REST API and most significantly a complete >>> HTML5 based web UI that will enable inspection of all QMF properties and >>> also queue/exchange/binding adding/deletion so that may be just the sort of >>> thing you're interested in - it's pretty close, I'm just doing some last >>> minute de-snagging trying to get it looking nice on small real-estate >>> mobile browsers. >>> >>> >>> To more directly answer your question "However, I’m having a hard time >>> making that same mapping between the connection stuff and a sender to an >>> exchange. " yeah there's some nuance there :-) the main thing is to think >>> in terms of UML associations, so for example a binding object has queueRef >>> and exchangeRef properties which hold ObjectIds that enable the associated >>> queue and exchange to be indexed (binding behaves like a UML association >>> class really) what this means though if you want to work out how a queue >>> links to an exchange you have to do it from the perspective of the binding. >>> >>> Similarly it's quite involved to find the subscriptions associated with a >>> queue because the navigation goes "the wrong way" that is to say >>> subscription has a queueRef property. >>> >>> What I've found is that in general the best (and by far in a way most >>> efficient) way to do non-trivial QMF stuff is to to the getObject() stuff >>> once for each of the objects that you care about then dereference those >>> objects into Maps indexed by objectId - if you do that then you can index >>> everything you want easily and cheaply. >>> >>> If you look at the code for things like qpid-config it's pretty easy to >>> get into the trap of doing getObjects() calls on inner loops, which is >>> heinously inefficient :-) because each getObjects() call does a messaging >>> request/response under the hood. >>> >>> To get you up and running and give a bit of insight (I hope!!) I've >>> attached a couple of python programs I wrote last year (these use QMF1) >>> >>> connection-audit checks when connections are made and looks up the queues >>> being subscribed to against a whitelist, basically if the queue being >>> subscribed to is not a queue that's in the whitelist it generates a log >>> message. >>> >>> connection-audit-dom is the same but uses DOM based XML parsing (I needed >>> to do that when I had to use it against an oldish python version) >>> >>> connection-logger does a lot of what I think you care about, basically it >>> logs all connections to a broker and provides useful info on them a la >>> qpid-config -b queues. >>> >>> >>> connection-logger-orig was my original version of this which was somewhat >>> erm "rushed out" :-) at face value it looks OK for a few queues etc. but if >>> you ramp up a load of connections - well try and add a couple of hundred >>> queues and you'll see what I meant by heinously inefficient above :-D. >>> >>> connection-logger and connection-logger-orig are functionally equivalent >>> so it's worth looking through both to give a bit of insight into the right >>> way and wrong way to go about things. >>> >>> I'm not a python programmer by any stretch of the imagination, but I think >>> that they are at least fair examples. >>> >>> Keep an eye out for my QMF2/UI update over the next few days but hopefully >>> this response has given you a decent leg-up. >>> >>> Best regards, >>> Frase >>> >>> >>> >>> >>> On 11/01/13 02:24, Lance D. wrote: >>> >>>> Hello all. >>>> >>>> >>>> I’m in the process of building QPID inspection tools for a project I’m >>>> working on. Now, I’m looking at the results of the qpid-tool. I see that >>>> the connection schema has a bunch of useful info (like remote pid, >>>> process, >>>> etc). I’ve also found ways to link that info up with the queues to figure >>>> out who’s subscribed to what. However, I’m having a hard time making that >>>> same mapping between the connection stuff and a sender to an exchange. >>>> >>>> >>>> >>>> My question is, is there a way to do that mapping (either with the >>>> existing >>>> tools, or by tapping into the qmf API? >>>> >>>> >>>> >>>> Thanks, >>>> -Lance >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> > > <example.cpp><test1.cpp><test2.cpp><UsingQMFv2MethodsToManipulateExchangesAndQueues.txt> > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected]
