On 03/02/14 15:38, Gordon Sim wrote:
Thanks for taking the time to write this up and share it with the
list! Feedback from users is what drives everything forward.
No worries, I've been meaning to find the time to properly explore the
AMQP 1.0 capabilities for ages. Part of my problem has been that there's
not a lot of concrete examples so it put me off somewhat, so I thought
that writing this might help someone else in the same position. I'll try
to tidy it up and put it into something like a PDF, the current flat
text isn't as readable as it could be. It'd be good to try and massage
some of this stuff into proper documentation.
* AMQP 1.0 specific message properties
The x-amqp convention is indeed a qpid::messaging specific approach to
handling fields that do not have explicit accessors in the Message
class. It is described in the AMQP_1.0 readme.
Yeah - to be honest the AMQP 1.0 readme started to make a whole lot more
sense *after* I started actually trying things out :-D I found myself
reading it, messing around for a while, then re-reading it. Some of the
stuff I've written down is me replying what's there as a way of trying
to externalise it in my own mind.
They should be set for outgoing messages as of 0.24[1]. What version
were you using?
I was actually using 0.27 off trunk built a couple of weeks ago.
As you note, the 'to' field can be set explicitly using the
'x-amqp-to' property name. There is also a connection level option
that controls whether that should be automatically populated with the
same address as used in the target of the attach (set_to_on_send,
which takes a boolean value).
I didn't notice that. Looking again in ConnectionOptions.cpp I can see
it now - though to be fair there aren't any comments that explain what
that option actually means, and I didn't notice it in the Doxygen
documentation.
http://qpid.apache.org/releases/qpid-0.24/messaging-api/cpp/api/classqpid_1_1messaging_1_1Connection.html
The 0-10 to 1.0 conversion currently maps the 0-10 message-transfer's
'destination' onto to the 'to' field. At the time I did it it seemed
logical enough, but I'm happy to remove that if it is confusing (as it
probably is).
It's a difficult call, I personally suspect that it's potentially
confusing. When I was trying this stuff out I wasn't sure if it was
something that AMQP 0.10 qpid::messaging did or was an artefact of the
mapping. It's probably safest to expect it to be explicitly added like
it is on Messenger, might be worth a wider discussion though.
* Text v. binary content
The translation from 0-10 messages to 1.0 equivalents only handles
content types for lists and maps at present. It should be improved to
check for text also. I've created a JIRA to track that[2].
Great, thanks!
* Subscription queues
I would say (loosely) that the subscription queue is part of the
terminus. I've added a sentence to the 1.0 readme describing how qpidd
constructs the subscription queue name.
What I couldn't figure was the relationship/association between node and
terminus. The AMQP 1.0 specification says "A link attaches to a node at
a terminus. There are two kinds of terminus: sources and targets" but
what's not especially clear in the specification is the multiplicity
between node and source or node and target. I think from what you are
saying here that there is a 1..* relationship. I knew it couldn't be
part of the link because of the ability to support shared subscriptions
(thank goodness!!).
I added the incoming and outgoing link entities as they seemed to be
useful when thinking in terms of the 1.0 model. The source and target
are as they appear in the attach received.
Yeah those seem to have potential, I stumbled across the "domain" and
"incoming"/"outgoing" stuff I've not had a chance to play yet but the
ability to establish links with arbitrary AMQP 1.0 containers seems
really cool. Is this stuff ultimately intended to replace the
"traditional" link/bridge federation model?
Though there is no QMF 'object reference' to the subscription queue
for an outgoing link from an exchange, the naming policy for
subscription queues does allow them to be mapped to their
corresponding outgoing links.
In QMF there isn't an object reference between an exchange and a queue
anyway, the association is via the bindings. It's really interesting to
look at this stuff through the GUI, what I see if I do:
./drain --connection-options {protocol:amqp1.0} -b localhost -f "amq.fanout"
Is a queue called <UUID>_amq.fanout_<UUID>
and if I look at the bindings I see
bind[] -> amq.fanout (and one to default direct)
So the exchange can be navigated to the queue and vice versa, but what
is missing is the QMF subscription object and without the subscription
object it's actually navigating between the subscription queue and the
connection (and vice versa) that's impossible - and I often find it
useful to see what Connection IPs are using which queues etc.
That said, I would agree that the situation around QMF and 1.0 links
can likely be improved upon, especially for people used to dealing
with the 0-10 model. I've raised a JIRA to ensure that doesn't get
lost[3].
So as above it's the missing subscription object that is the main issue
I was flagging, but it would indeed be really nice to have the
associations with incoming/outgoing links be navigable.
* Reply-to
The qpid::messaging library doesn't particularly restrict the value of
the reply-to in the message itself[4].
If it can be interpreted as of the form <name>/<subject>, then the
address object returned will have both the name and subject set to the
respective substrings. Otherwise the name will contain the full reply-to.
If the application takes that address and uses it directly as an
address for creating a sender, the name is used as the source or
target address for the link and if a subject is specified it will be
used to set a filter.
I believe a common usage will have reply-to simply contain a node name
(often for a temporary queue). That pattern, essentially the
client-server example, works against several brokers and also the
dispatch router[5]. It is also familiar from JMS.
However if some other specific scheme/format is used, the application
would responsible for interpreting the address and e.g. establishing
separate connections etc if that behaviour is desired.
I'll need to have more of a play with Reply-To but the thing I was
really trying to get at was that given the peer to peer model of AMQP
1.0 it is (presumably) possible to set a Reply-To to the complete unique
address of the sender (including host:port part) and the receiver could
presumably directly address the sender without necessarily returning via
intermediaries. That scenario seemed awkward for a connection oriented
paradigm such as qpid::messaging or JMS because the host:port part might
be an entirely different connection to the one it has used to connect to
say a broker.
You are probably right that having the reply to refer to a node name is
likely to be a common usage pattern for client/server style applications
I was mainly musing about what a "fully qualified" address might mean
for connection oriented paradigms - if you see what I mean.
* Shared subscriptions
The 'shared' subscription capability is described in the AMQP 1.0 readme.
Yeah I did notice it there, my problem was working out what the syntax I
needed to use was.
Note that it is a qpidd specific extension. At present the
subscription queue will be autodeleted unless you set the link to be
durable or reliable.
Ah I didn't realise that the reliable flag would help me. So would that
be something like reliability:|||at-least-once| in the link properties?
So are you saying that other AMQP 1.0 brokers generally don't support
shared subscriptions? TBH that seems unfortunate I know that I find them
incredibly useful to distribute workload between physical consumers.
Supporting an expiry policy of 'never' should be possible as well.
I'll update the test to note the convention around queue name.
What I was getting at in that section of my post was "I couldn't see
anything in the qpid::messaging code related to expiry" in other words
there was no reference to pn_terminus_set_expiry_policy so (I think)
it's additional code that's needed - have I missed something?
The auto-delete value specified on a topic will now be correctly used
for subscription queues created for that topic[6].
Great, thanks - at the very least it will make things "feel" consistent
having that ability.
You can also specify a lifetime-policy e.g. to delete only when not
used *and* empty. This is a nice option as it will be recreated if
ever needed, but won't sit around taking up resources if not.
Ahh I didn't realise that you could set the lifetime-policy for topics,
I assumed that it was specific to node creation.
* The 'create' option in qpid::messaging addresses
Using this option with an explicit node name results in non-standard
AMQP and is discouraged where it can be avoided.
Yeah I realised that , I was just playing :-) Is there a reason for
that (other than it being what the AMQP 1.0 specification says) not sure
why the spec. chose to say this?
A better alternative is to configure node policies. These are patterns
that an attaching links address are checked against if it doesn't
resolve to any existing node. If a matching pattern is found, the node
will be auto-created with the properties of that policy.
Ahhh, again that's something that I hadn't come across, I noticed
QueuePolicy/TopicPolicy in the QMF managemment-schema.xml and it was
bugging me what they were for.
* Filters
In the case of the headers exchange, the key could be taken to be the
name of the filter, as the key is defined simply to be an identifier
for that exchange. However for other exchanges this would not be the
case (e.g. for topic, direct and even the qpidd specific xml
exchanges, the key affects matching and is therefore in my view
logically part of the filters value).
Yeah I'd agree for topic and direct the key relates to the value, but in
the case of headers exchange there does need to be a way of setting the
binding key and it seems (well to me) that the filter name is probably
the most convenient thing.
As I say in my post I have scenarios where I might have multiple
bindings between the headers exchange and a queue and these all need
different keys. At the moment the binding is [] so the last specified
filter is the only one that actually gets applied.
Note also that in AMQP 1.0 a message must match *all* the specified
filters, so you can't emulate OR with a set of different filters.
So that's slightly worrying I hadn't clocked that. What I'm not clear
about then is how I'd go about mapping the scenario I had with:
x-bindings: [{exchange: 'amq.match', queue: 'testqueue', key: 'key1',
arguments: {x-match: all, data-service: amqp-delivery, item-owner:
fadams}}, {exchange: 'amq.match', queue: 'testqueue', key: 'key2',
arguments: {x-match: all, data-service: http-delivery, item-owner: fadams}}]
In this AMQP 0.10 case I've got two separate bindings added between
amq.match and "testqueue" so messages will be delivered if either of
them match.
I (naively) thought that multiple filters was the way (though on
reflection I can see that all filters must match - how else would my
topic plus selector example work the way it did d'Oh).
Can you think of a way how the use case I've suggested might be
supported (aside from selectors - they are cool, but it'd be good to be
able to replicate the legacy headers stuff completely - especially for
migration/integration purposes).
For the xml exchange the binding key is in fact an additional
filtering element that must match the subject. It is not just an
identifier, as it is in the case of the headers exchange. You can set
the key using the 'subject' in the address, e.g. xml/weather.
Ahhh I think I'm getting you now, I've just tried:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"xml/weather; {link: {name: test-link, filter: {name: 'weather',
descriptor: 'apache.org:xquery-filter:string', value: \" \
./weather/station = 'Raleigh-Durham International Airport (KRDU)' \
and ./weather/temperature_f > 50 \
and ./weather/temperature_f - ./weather/dewpoint > 5 \
and ./weather/wind_speed_mph > 7 \
and ./weather/wind_speed_mph < 20 \
\"}}}"
and that does indeed set weather as the binding key, thanks.
Fair play though I thought that I did pretty well getting as far as I
did with the filter stuff :-D
The version in the "Programming" book says:
./drain -f "xml; {link:{x-bindings:[{key:'weather',
arguments:{xquery:\"$(cat rdu.xquery )\"}}]}}"
So, setting the key rather than the subject.
To be fair I actually think using the subject is clearer, but you can
hardly blame me for getting a bit confused.
Although AMQP itself does not place any restrictions on application
property names, the selector syntax for AMQP 1.0 is that of JMS
selectors[7], where as you noted, data-service is not a valid name.
The extension of the selector explicitly states "JMS header names
should be translated to amqp.<field_name> where <field_name> is the
appropriate AMQP 1.0 field named in the table above, with the hyphen
replaced by an underscore."
However we could probably look at ways to (optionally?) make this more
lenient. It certainly is also something that should be highlighted
more prominently in documentation somehow.
It would be *really really good* to be able to make this more lenient
(would quoted field names be a possibility e.g.
'data-service'='amqp-delivery')?
I've used hyphened property names quite a lot and not being able to use
them would be a bit of a show stopper for me due to the integration
issues (I've got a lot of producers and they certainly couldn't all be
changed simultaneously) and given the apparent support at the moment for
multiple headers bindings between amq.match and a given queue I'm a bit
stuck if I want to migrate.
Re: "The extension of the selector explicitly states "JMS header names
should be translated to amqp.<field_name> where <field_name> is the
appropriate AMQP 1.0 field named in the table above, with the hyphen
replaced by an underscore."
True, but that could be interpreted to mean the *standard* JMS header
names, and being slightly flippant it also says "The "properties" of the
JMS message are equivalent to the AMQP application-properties section" -
which simply says string.
It's a bit moot, I'm just having a bit of fun around how one can
interpret things, but I really would be grateful if you guys could have
a think about being able to support hyphened property names (clearly
there's a potential conflict with the minus sign which is why I was
wondering if quoting the property name was an option).
As I say for my current set of scenarios it makes things rather awkward :-/
I believe the selector filters are supported by a number of brokers
outside Qpid (ActiveMQ, HornetQ, SwiftMQ).
That's useful to know, have you tried many interoperability scenarios?
However the legacy-amqp-filters are only supported by the two Qpid
brokers at present as far as I am aware.
That's not such a big deal at the moment, I'd definitely look to move to
a subject plus selectors model if I can, that last example I did was
really cool and selectors are so expressive. I'd have given my *eye
teeth* to have had selectors when I first started using Qpid - when I
was migrating data flows from the previous system we had having a "NOT"
would have been soooooo useful.
[1] https://issues.apache.org/jira/browse/QPID-4707
[2] https://issues.apache.org/jira/browse/QPID-5536
[3] https://issues.apache.org/jira/browse/QPID-5537
[4] https://issues.apache.org/jira/browse/QPID-5168
[5] https://issues.apache.org/jira/browse/DISPATCH-1 and/or the
set_to_on_send connection option
[6] https://issues.apache.org/jira/browse/QPID-5469
[7] "The selector-filter uses the selector as defined by JMS."
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
BTW were my observations that Proton Messenger isn't currently able to
support the more sophisticated subscription patterns correct? If so is
this something that is likely to be addressed (no pun intended) :-)
Many thanks for reading through this and responding so quickly.
Frase
