See bug 794802 for some of the gory details; in short bugsummary managed to spring a nasty surprise on us.
We decided to roll forward given a few factors - the problem is isolated (though to a substantial fraction of the system), its tricky to change live no matter what, and we didn't know what the precise cause was - without know that changing stuff in a panic wouldn't necessarily fix it.. because its a schema change its not simple to rollback. We didn't want to make things worse by poking haphazardly without Stuarts help either, (so we got that :)) Much of the delay in fixing was our 4 hour test suite validating the key change to fix the dirty-great-lock; thats fixed (Thanks stub), but there was/is a hidden issue behind that - the UPDATE was doing table scans due to the use of IS NOT DISTINCT FROM, which made the implementation short and sweet but also made the query pathological. So we easily added some seconds or more (at 0.3 seconds per dimension combination) to bug updates, and with bug heat updates in Ubuntu were slow anyway. At this point I've handed over the incident to Stuart, and am halting() myself. I hope to awaken to great news :) -Rob _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
