Thanks Ken,
I definitely think it's a far better proposal to provide an API
extension (objectUpdate) than rather than something which is confusing
at best (and more likely a breaking change that will probably break in
subtle ways).
One thing though, I think that we discussed on the main thread (and Bill
mentioned this too) about providing an extension to specify the QMF
version to consume. As it stands it'll subscribe to
qmf.default.topic/data.ind.* or something like that for QMF1 and a
similar subscription off (I think) qpid.management for QMF2 pushing
essentially the same data twice is somewhat wasteful of bandwidth. There
*may* be applications that need to deal with a mixed economy (I'm not
convinced that would be the case from a single connection/console
instance but maybe that's just me) but in any case as we have discussed
"Its likely that apps written using the console haven't ever received v2
style updates" so I'm tempted to suggest that (despite wanting to move
away from QMF1!!) the optimal non-breaking change would be to default to
subscribing to qmf.management and delivering to objectProps/objectStats
with additional API calls to enable QMF2 only and to enable both (I'm
sceptical about this last use case but if it doesn't harm anything else
I don't mind giving the choice).
For most users it might not be crippling to get the same data twice, but
it's unwise to presuppose application use cases, I was messing around
once with a test case covering a fairly large number of consumers (1000
connections, the same for queues and a few thousand bindings) I hadn't
really thought much of it previously until I noticed that my MIS traffic
was taking way more bandwidth than my real traffic (I was monitoring
queue, exchange and connection traffic) given that the QMF2 data are
presented as Maps IIRC with a 10 second data interval I was getting MB/s
:-D (ever come across the Heisenberg Uncertainty Principal :-) I
definitely assert that it applies as much to data management as quantum
physics ;->)
Actually that reminds me, you said "V2 handles these updates in a
different way: it is optimized to generate only one update that combines
both statistics and properties " - are you sure?
It has been over a year since I looked at this (I mostly do synchronous
getObjects() calls) so my recollection is slightly vague but you may or
may not know this but the QMF2 API spec has the notion of "query
subscriptions" which are quite neat. When I implemented this in the Java
implementation of that API I hit a teeny weeny hurdle - basically the
C++ broker Management Agent doesn't actually support that part of the
QMF2 protocol :-D what I did when a query subscription was specified
against the broker Management Agent was to set up a (AMQP/JMS)
subscription to qmf.default.topic/data.ind.* then apply the query
specified as a filter. It was sub-optimal as the data is still being
sent over the network, but from an application perspective it works
pretty well (my Java QpidQueueStat implementation actually used this
approach kind of as a test case and as an API demo).
Anyway... and as I say it has been a little while. One thing that I seem
to recall was that if my query subscription was based on properties all
was well, but if it included statistics it wasn't doing what I was
expecting. From memory what I *think* happens is that "one update
combines both statistics and properties" indeed, but that update only
contains any statistics that have actually changed. It's not *quite* the
same thing as saying "update that combines both statistics and
properties" users of this API will still have to have code very similar
(but not quite the same) as they'd have needed for V1 in order to "join"
and present the complete representation of the QmdData instance.
I'll grant you it's more bandwidth efficient (though I suspect only
marginally given that most stats on the most frequently changing Objects
will be changing) but it's one heck of a lot more confusing to users
behaving the way it does.
As I'm typing my memory is coming back and I'm *pretty sure* that this
is indeed how QMF2 unsolicited data updates are currently working.
Frase
On 10/04/13 16:41, Ken Giusti wrote:
Folks,
Recently I submitted a fix on trunk that addressed a problem where QMF v2
object update notifications were being dropped by the python QMF console
(qmf.console). The fix merely re-enabled the update notifications.
As pointed out by both Fraser Adams and Bill, this may cause problems with
existing clients. As described below (and elsewhere in this thread), now both
QMFv1 updates and QMFv2 updates will be delivered via the qmf.console.Console
callback interface.
Sadly, the way this is done is confusing at best. For V1 updates, there are
separate callbacks for statistics updates (via objectStats()) and property
updates (via objectProps). The confusion comes from the fact that V2 handles
these updates in a different way: it is optimized to generate only one update
that combines both statistics and properties. However that update arrives via
the objectProps() callback. For QMFv2, the objectStats() callback is ignored.
This forces the application's objectProps to have to deal with different update
formats - v1 and v2 - and handle the different use cases (updating statistics
in the v2 case, not for v1).
AFAIK - this bug has prevented v2 updates from arriving on that objectProps
callback since prior to 0.18. Its likely that apps written using the console
haven't ever received v2 style updates.
I propose we fix this in the manner suggested by Bill and Fraser. Specifically:
1) Add a new interface to the qmf.console.Console object:
def objectUpdate( self, broker, record )
This callback will be invoked only for QMFv2 object updates. The parameters
are analogous to objectProps and objectStats
2) The existing objectProps and objectStats callbacks are invoked only for the
v1 style updates, as was their behavior prior to my fix. v2 object updates
will never arrive on these callbacks.
Any objections?
----- Original Message -----
From: "Bill Freeman" <[email protected]>
To: [email protected]
Sent: Friday, April 5, 2013 4:20:37 PM
Subject: Re: Questions from a novice
On Fri, Apr 5, 2013 at 3:38 PM, Ken Giusti <[email protected]> wrote:
----- Original Message -----
From: "Bill Freeman" <[email protected]>
QMF V2 has been available for some time now, and we've cut all the project
tools and libraries over to speaking it. If we start down the road to
deprecating V1, turning off the broker updates is about the least risky
first step I can think of. It would not remove v1 at all - but it would
require manual intervention to restore the old behavior. Having to
explicitly take that action would be a (hopefully gentle) nudge to assess a
deployment's V1 dependency.
I think the on line documentation may have something to answer for here.
When I started this project, since we are using MRG, I started with
RedHat's documentation, but didn't find any advice about coding QMF. Then
Google got me to the Apache Qpid pages, link to QMF, and eventually
dribbled down to an example that became the base of my current design.
That was based on qmf.console. There was no insight there about selecting
V1 versus V2. There were also several pages, at least, that seemed to
imply that V1 was now, and everything would support it, while V2 was new
and for the future, and not universally supported (and that turned out to
be correct until your patch).
I think that the concept that consoles, for some time yet to come, should
endeavor to support both versions, is a good one. They should perhaps log
a deprecation warning if they discover a V1 only broker (aren't V2 brokers
supposed to translate for V1 only agents?), but they should work anyway.
One thing that makes it tough for consoles to do so, at least if they are
based on qmf.console, is the fact that the objectProps callback is shared
between versions, but has a different meaning. One callback carrying
deltas is a good design, but it seems that either it should use a
completely separate callback, or it should call both objectProps and
objectStats (in that order), omitting objectProps if it has no properties,
and omitting objectStats if it has no statistics. A V2 specific callback
would be my preference, but only by a little. Since the other stuff has
never been called for V2 updates before, this is the time to make that
change.
And I guess that's also a potential problem with making the fix from your
patch: Code that has been working for years using qmf.console has never
received a V2 objectProps call before may not work with it (mine certainly
didn't). A separate callback for V2 updates fixes this issue.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
