Hey Bill, I've cc'ing Fraser and the rest of the dev team on this - been meaning to raise this as a general issue with the C++ broker.
I'm trying to find the best way to fix these problems, and I've started making changes on trunk. You've been bitten by my latest attempts - sorry. I first started to implement the additional objectUpdate callback as originally proposed. Very easy to do. But that additional api required other tools to be updated - and additional documentation changes, etc. After implementing this, I discovered a problem with the fix: tools that need to monitor both QMFv1 and QMFv2 agents starting reporting incorrect results (qpid-tool, to be specific). These tools need to support both the old callbacks (for QMFv1 updates) and the new one (for QMFv2 updates), in order to see both QMFv1 and QMFv2 agents. And that's when I realized that this fix was just a hack to work around the actual problem. The real issue at hand is that a QMF agent must not transmit both QMFv1 and QMFv2 updates for the same object. That's really what's broken here, and the C++ broker is doing precisely that. Why is that a problem? Because a console receiving these two updates cannot tell that they are for the same object. To the console application, it appears as two separate objects. Yep - QMFv1 object identifiers have no relation to QMFv2 object identifiers, by design. Any attempt to compare the two will result in inequality, even if the underlying object is the same. Take a look at qpid-tool on trunk. Run it against a broker. It will end up reporting twice as many queues, exchanges, etc, and there really are. This is because the tool stores the objects by their object ids - which is precisely what it should do. But the broker is sending two updates for each object, and now that the console is no longer dropping the QMFv2 update, its adding those objects to its database. Since the object ids for the two updates don't match, the database stores each as a separate object. I'm proposing that we should solve this problem by disabling QMFv1 updates coming from the C++ broker. This should be the default for the next release (0.26). It should still be possible to turn them on manually if desired, but the C++ broker should only transmit QMFv2 type updates going forward. We can minimize the impact to console applications by having the objectProps and objectStats callbacks be invoked when a QMFv2 object arrives instead of introducing a new callback. [Bill - I'll fix the console to not call objectProps when only stats are present]. I think this should eliminate the need to re-code console applications. This will cause problems for folks that use an old python console against the next release of the broker. These folks will have to upgrade the console as well. Does anyone have a better alternative? -K ----- Original Message ----- > From: "Bill Freeman" <[email protected]> > To: "Ken Giusti" <[email protected]> > Sent: Thursday, April 25, 2013 5:07:44 PM > Subject: Re: QMFv2 object update bug [WAS Re: Questions from a novice] > > On Wed, Apr 10, 2013 at 11:41 AM, Ken Giusti <[email protected]> wrote: > > > Folks, > > > > Recently I submitted a fix on trunk that addressed a problem where QMF v2 > > object update notifications were being dropped by the python QMF console > > (qmf.console). The fix merely re-enabled the update notifications. > > > > As pointed out by both Fraser Adams and Bill, this may cause problems with > > existing clients. As described below (and elsewhere in this thread), now > > both QMFv1 updates and QMFv2 updates will be delivered via the > > qmf.console.Console callback interface. > > > > Sadly, the way this is done is confusing at best. For V1 updates, there > > are separate callbacks for statistics updates (via objectStats()) and > > property updates (via objectProps). The confusion comes from the fact that > > V2 handles these updates in a different way: it is optimized to generate > > only one update that combines both statistics and properties. However that > > update arrives via the objectProps() callback. For QMFv2, the > > objectStats() callback is ignored. This forces the application's > > objectProps to have to deal with different update formats - v1 and v2 - and > > handle the different use cases (updating statistics in the v2 case, not for > > v1). > > > > AFAIK - this bug has prevented v2 updates from arriving on that > > objectProps callback since prior to 0.18. Its likely that apps written > > using the console haven't ever received v2 style updates. > > > > I propose we fix this in the manner suggested by Bill and Fraser. > > Specifically: > > > > 1) Add a new interface to the qmf.console.Console object: > > def objectUpdate( self, broker, record ) > > This callback will be invoked only for QMFv2 object updates. The > > parameters are analogous to objectProps and objectStats > > > > 2) The existing objectProps and objectStats callbacks are invoked only for > > the v1 style updates, as was their behavior prior to my fix. v2 object > > updates will never arrive on these callbacks. > > > > Any objections? > > > > > I may be cross eyed, but it appears to me, in the latest in subversion > checkout, that something different happened. (I'm getting ready to deploy, > and want to make something I can install with pip, so I'm looking to work > with the latest and use v2, so that led me to poke in subversion.) > > Rather than v2 updates calling session.concole.objectUpdate() only, there > is still no session.console.objectUpdate() call, and a v2 update that > contains statistics will also call both session.console.objectProps() and > session.console.objectStats(). See around line 3112. > > (As to subsequent updates, they might only go to objectStats(), though I > doubt it, since, after the patch in _tryToConnect(), I never saw a v2 > update call objectStats(), only objectProps(), so I'm guessing that v2 > updates always have qmf_object in the keywords, and never qmf_object_stats > .) > > With this approach objectProps() will still be getting both v1 and v2 > calls, and will sometimes get v2 calls without properties. I believe that > any unknown existing code out there is still likely to break if they > upgrade to this qmf.console. That would not be true with Ken's suggested > approach above. Old code, never having heard of the objectUpdate() console > callback method, won't have a handler for it, and the v2 callbacks would be > ignored, rather than causing trouble. > > Was this, perchance, because there is within qmf.console.Agent.__call__, > some expense in determining whether an update is v1 or v2 (instantiating > the ObjectID)? Because I'm going to wind up doing that anyway, since I > need to discard one set of updates or the other. > > (Isn't anyone using this API going to need the ObjectId instance anyway, to > be able to figure out to which object this update applies? If it's > expensive to instantiate the ObjectId, then it's actually a pity that > Agent.__call__ doesn't always do it and pass it to callbacks, allowing it > to only be done once per update.) > > Bill > -- -K --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
