Hi Michal
Epic TL;DR response follows - I'm afraid I got on a roll ..........
You are using QMF to talk to the c++ broker (qpidd) right?
Presumably when you say "the QMF library delivers this update with
ObjectId of the form..... " you are doing this via the getObjects()
call? E.g. something like:
List<QmfConsoleData> queues =
_console.getObjects("org.apache.qpid.broker", "queue");
Is that correct?
If so the code for that is in
qpid/tools/src/java/qpid-qmf2/src/main/java/org/apache/qpid/qmf2/console/Console.java
(line 697 on trunk currently).
As far as I can recall that code simply reflects the information in the
QMF object (e.g. the queue objects in your case) that was sent by the
broker. It receives the message from JMS via:
Message response =
_responder.receive(timeout*1000);
then subsequently puts the information into a List<QmfConsoleData> via
if (AMQPMessage.isAMQPList(response))
{
List<Map> mapResults =
AMQPMessage.getList(response);
partials.ensureCapacity(partials.size() + mapResults.size());
for (Map content : mapResults)
{
partials.add(new
QmfConsoleData(content, agent));
}
}
That code caters for the fact that JMS doesn't support lists terribly well
qpid/tools/src/java/qpid-qmf2/src/main/java/org/apache/qpid/qmf2/common/AMQPMessage.java
provides some code to make it a bit easier to deal with Map and List
messages in java.util Containers.
It basically first decodes the JMS message as a List<Map> because QMF2
query responses get sent by the Broker as an AMQP List of AMQP Maps. I
then go through and create a List<QmfConsoleData> where the
QmfConsoleData is basically a wrapper around a java.util.Map
I've looked at QmfConsoleData and when that gets constructed or
initialised the ObjectId instance _object_id (which is actually owned by
the parent QmfDescribed class) gets set via
new ObjectId((Map)m.get("_object_id"));
Which is explicitly getting the property named "_object_id" from the Map
used to construct the QmfConsoleData, which is as I say a property that
has been set in the QMF2 Message sent by the broker.
So in a long, roundabout, thinking out loud nutshell I'm pretty sure
that the "particular reason why the Java implementation of QMF doesn't
generate more unique ObjectId" is because it's not the Java
Implementation of QMF that is actually generating the ObjectId in the
first place, it's simply representing exactly what was sent to it by the
Broker.
You mention that you've also been using the Python QMF implementation,
can I ask which one? There are actually two Python QMF implementations
there's one that lives in:
qpid/extras/qmf/src/py/qmf/console.py
and is used by things like qpid-queue-stats and a few other things. This
is the asynchronous API and starts something like:
class BrokerManager(Console):
The other python API lives in
qpid/qpid-trunk/qpid/tools/src/py/qpidtoollibs/broker.py
And that's the one used by the python qpid-config etc.
Looking in broker.py getAllQueues calls getAllBrokerObjects which calls
_doClassQuery which does
items = []
done = False
while not done:
for item in response.content:
items.append(item)
Which again appears pretty much to be reflecting what is being sent by
the broker.
So when you say you've used Python's QMF implementation I'm suspecting
that you aren't using getAllQueues from qpidtoollibs/broker.py. but are
in fact using qmf/console.py?
(sorry this is now really TL;DR but I tend to think out loud and the
play back is I think useful - my Maths lecturer always drummed into me
"show your working" so it's all his fault :-))
Right....... I've now just proved to my satisfaction that it is indeed
the behaviour of the Broker. What I've just done is gently hacked the
qpid-config port that I've just written in JavaScript, that's a good
"control" because it's independent of either the Java QMF or
qpidtoollibs and also uses AMQP 1.0 and proton Messenger so should rule
out any accidental bias.
What I did was hack the queueListRecurse method to look like this:
for (var i = 0; i < queues.length; i++) {
var queue = queues[i];
var queueId = oid(queue._object_id);
console.log("queueId = " + queueId);
.......
So I'm basically printing out the ObjectId that I get directly from the
QMF response. In my console I see....
<start qpidd here>
./qpid-config.js -r queues
queueId =
1815:org.apache.qpid.broker:queue:4c44dfde-5ea6-4370-d102-549237ac4ac7_#
Queue '4c44dfde-5ea6-4370-d102-549237ac4ac7_#'
bind [4c44dfde-5ea6-4370-d102-549237ac4ac7_#] => ''
<that's the oid of the temporary QMF response queue BTW. In another
window I do qpid-config add queue test>
./qpid-config.js -r queues
queueId =
1815:org.apache.qpid.broker:queue:b5250fd9-80fc-4208-94e0-8e1bbc41106d_#
Queue 'b5250fd9-80fc-4208-94e0-8e1bbc41106d_#'
bind [b5250fd9-80fc-4208-94e0-8e1bbc41106d_#] => ''
queueId = 1815:org.apache.qpid.broker:queue:test
Queue 'test'
bind [test] => ''
<So now you can see the oid of the "test" queue. In the other window I
then do qpid-config del queue test>
./qpid-config.js -r queues
queueId =
1815:org.apache.qpid.broker:queue:5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#
Queue '5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#'
bind [5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#] => ''
<as you can see, it's gone again. Now I do qpid-config add queue test again>
./qpid-config.js -r queues
queueId =
1815:org.apache.qpid.broker:queue:f7e8307e-c8ce-441c-8279-f543ad91b076_#
Queue 'f7e8307e-c8ce-441c-8279-f543ad91b076_#'
bind [f7e8307e-c8ce-441c-8279-f543ad91b076_#] => ''
queueId = 1815:org.apache.qpid.broker:queue:test
Queue 'test'
bind [test] => ''
Clearly queueId = 1815:org.apache.qpid.broker:queue:test is back and has
essentially replicated your observation, but it's demonstrably yielding
what the Broker is telling it and is not a case of the QMF library
(Java, qpidtoollibs or the JavaScript) itself generating any ObjectID.
What I *think* you are observing in the qmf/console.py library is
possibly something to do with the old QMF1 messages (IIRC the Broker can
send both formats) the old QMF1 format was a binary form and the OIDs
were IIRC quite different in structure (I'm afraid I don't know too much
about QMF1).
Now that I think about it (sorry it's taken a while until the penny
finally dropped) the QMF2 specification
(https://cwiki.apache.org/confluence/display/qpid/QMFv2+API+Proposal)
actually says something about ObjectIDs and allows them to be "predictable"
"
An object identifier uniquely addresses a data object within the domain
of its managing agent. QMF represents the object identifier as an opaque
string. An object identifier can be assigned to the object by the agent
when the object is instantiated.
Alternatively, a schema can define an object identifier by defining an
ordered list of names of data items. In this case, the object identifier
string is built by concatenating the string representations of the value
of each named data item. This approach is similar to defining index
fields within a database record.
"
It's the second paragraph that is in fact important for the purposes of
your observations "concatenating the string representations of the value
of each named data item" - does this ring any bells:
1815:org.apache.qpid.broker:queue:test
TBH I should have twigged sooner, I've actually been bitten by this in
the opposite direction. As it happens I implemented a QMF2 plugin for
the JavaBroker a while back and I actually originally implemented the
ObjectIDs by assigning UUIDs, but when I started testing it I noticed it
didn't work with the Python qpid-config, though it *did* work with my
Java qpid-config port...
I eventually figured out that my Java and JavaScript qpid-configs make
no assumptions about the ObjectID, so in order to invoke CRUD methods
(like in qpid-config add queue test) they query for the "Broker"
management object and invoke methods on that, but what qpidtoollibs does
is the following:
"
def _method(self, method, arguments,
addr="org.apache.qpid.broker:broker:amqp-broker", timeout=10):
props = {'method' : 'request',
'qmf.opcode' : '_method_request',
'x-amqp-0-10.app-id' : 'qmf2'}
correlator = str(self.next_correlator)
self.next_correlator += 1
content = {'_object_id' : {'_object_name' : addr},
'_method_name' : method,
'_arguments' : arguments}
message = self.message_class(
content, reply_to=self.reply_to, correlation_id=correlator,
properties=props, subject="broker")
self.tx.send(message)
response = self.reply_rx.fetch(timeout)
self.sess.acknowledge()
if response.properties['qmf.opcode'] == '_exception':
raise Exception("Exception from Agent: %r" %
response.content['_values'])
if response.properties['qmf.opcode'] != '_method_response':
raise Exception("bad response: %r" % response.properties)
return response.content['_arguments']
"
In other words it uses "org.apache.qpid.broker:broker:amqp-broker" which
is actually the ObjectID of the Broker Management Object, so
qpidtoollibs is choosing not to query for the Broker Management Object
to retrieve its ObjectID because it knows that the C++ Broker has
implemented ObjectIDs according to the second paragraph I quoted above.
So in precis:
1) The Java, Python qpidtoollibs and JavaScript QMF2 code are all
behaving as expected and are not in fact generating their own ObjectIDs
they are using what the Broker sent them.
2) The Broker is indeed behaving in the way that you observed, but is in
fact behaving correctly according to the QMF2 Specification.
Sorry it took so long to reach a conclusion that you probably didn't
want to hear, but hopefully the journey to the conclusion was
enlightening - and at least shows I'm not making it all up :-D
Sorry this probably doesn't help with your problem. I'm a bit torn - as
I say I originally implemented OIDs using UUIDs and changed the
behaviour to be consistent with the C++ Broker and qpidtoollibs, but
OTOH I suspect that for many people (if not the vast majority -
excepting you obviously) If you create a queue called "test" it's
probably actually quite useful to be able to refer to it by name
org.apache.qpid.broker:queue:test and if you delete it and re-add it
there's likely a reasonable argument either way as to whether you might
want it to be the *same* Object (e.g. the same object name in the same
logical namespace such as a queue and this approach gives your IDs
temporal persistence even surviving Broker restarts) the other side of
the coin is that having them unique is also not necessarily an
unreasonable assumption too - but that I'm afraid is not the way it's
interpreted by the Broker, and making them unique would itself break
qpid-config and most likely a bunch of other QMF tools, so it's probably
not going to gain much traction trying to push for a change in behaviour.
Re "It is quite important for us, since we want to keep track of also
deleted objects on our side. However, if the QMF library gives us object
with the same ObjectId (which we already keep as deleted) we have a
collision. "
I think TBH that you are simply going to have to change your algorithm.
In practice if you delete a queue named test_queue then re-add the queue
test_queue you've basically restored exactly the same QMF object as far
as the Broker is concerned, so technically you don't have a collision
you have an ObjectID that refers to the same Object and the deletion of
test_queue was merely transient behaviour.
Now if you actually *care* that the queue has been temporarily deleted
(perhaps for auditing purposes or whatever) then you have a few options
- but as far as the Broker is concerned they are the *same* Object.
If you want to identify that this has occurred you need to bear in mind
that although you say "we are using Java QMF library for receiving
asynchronously broker's objects updates " if you are using the
getObjects() stuff you are not actually working asynchronously you are
infact polling.
To deal more asynchronously you could either use QMF2 Events (which are
delivered asynchronously) and either use the info from the Event or use
that to trigger a call to getObjects() - look in
qpid/qpid-trunk/qpid/tools/src/java/qpid-qmf2-tools/src/main/java/org/apache/qpid/qmf2/tools
The ConnectionAudit.java and ConnectionLogger.java examples both
basically receive QMF Events
if (wi instanceof EventReceivedWorkItem)
.....
Then subsequently go on to do a bunch of getObjects() calls, so you
would see IIRC queueDeclare/queueDelete events that would relate to your
"test_queue"
Alternatively you could make use of the QMF2 API "Query Subscription"
mechanism. If you look in QpidQueueStats.java I've given an example of
how to use that, now I *personally* wouldn't go down that route.
Although my Java QMF2 API has implemented everything specified in
https://cwiki.apache.org/confluence/display/qpid/QMFv2+API+Proposal (as
you may have gathered I'm just a bit OCD) as it happens the Java QMF
implementation is actually the only API that ever actually got round to
implementing all of that...... what basically happened was the
realisation that the QMF2 protocol is really when all is said and done
just a bunch of Map Messages, so there's a bit of an argument whether
one should use an API or simply just use the Map Messages. My view is
that there's more of an argument for an API in strongly typed languages
like Java and C++ (most of QmfConsoleData is actually defensive code
that I discovered the hard way 'cause qpidd sends strings as AMQP byte
and I got killed by ClassCastException when I tried to cast byte[] to
String :o)). It's also woth pointing out that there's likely to be a (I
suspect vaguely slow) migration from QMF to AMQP 1.0 Management
Specification and at some point when I get my act together I'll probably
try to provide a mechanism to get info from both protocols to ease the
transition, at which point I *might* deprecate some of the more
fancy/fiddly features of the QMF2 API 'cause I suspect that nobody has
actually used them since I wrote that demo...... If you stick to
getObjects and the Event stuff I'm more likely to try to make that
polyglot than the Query Subscription stuff, which was flipping fiddly to
implement if I'm honest.
I really hope that this has been useful even if not quite what you'd
hoped for, sorry it ended up as "War and Peace", but as a result you now
probably know about as much as anyone does on QMF :->
Finally, although I don't mind answering PM'd questions I'd be grateful
if you could post any other qpid related questions to the user list -
it's mainly because there's a fighting change that if you have a Qpid
related question you won't be the only one who might like to know the
answer.
Cheers,
Frase
On 17/09/14 12:07, michal.zer...@gmail.com wrote:
Hello Fraser,
we are using Java QMF library for receiving asynchronously broker's objects
updates. I have a question regarding the ObjectId - which is and object
identifying the broker's object change.
Imagine I create a queue test_queue on the broker, the QMF library delivers
this update with ObjectId of the form (toString()):
@2...@org.apache.qpid.broker:queue:test_queue
Now I delete the queue and I get the update identified again with the same
ObjectId. So far so good. But if I create the queue again (with the same name),
QMF library sends me the update again with the same ObjectId. This is different
comparing to e.g. Python's QMF implementation, where I would get the new ID.
It is quite important for us, since we want to keep track of also deleted
objects on our side. However, if the QMF library gives us object with the same
ObjectId (which we already keep as deleted) we have a collision.
So questions from my side:
- Is there a particular reason why the Java implementation of QMF doesn't
generate more unique ObjectId (like Python implementation does)?
- Would it be possible to change this behavior? We can implement it by
ourselves and provide the patch to the Qpid community if you can give us brief
instructions on it.
Thank you in advance!
Best regards,
Michal
_____________________________________
Sent from http://qpid.2158936.n2.nabble.com
