On Fri, Jun 10, 2011 at 4:19 PM, Robert Collins <robe...@robertcollins.net> wrote: > On Thu, Jun 9, 2011 at 10:44 PM, Robert Collins > <robe...@robertcollins.net> wrote: >> See bug 794802 for some of the gory details; in short bugsummary >> managed to spring a nasty surprise on us. > ... >> At this point I've handed over the incident to Stuart, and am halting() >> myself. >> >> I hope to awaken to great news :) > > So this is basically behind us - there is a cowboyed schema change > live that preserves performance - the branch for it to exist > officially will be landing on devel soon. Until that lands the tags > portlets for projects won't adjust (specifically closing all the bugs > for a tag won't remove that tag). > > Some observations: > - we had a 900 timeout spike over the period of degraded service. > Thats not too bad - its just over one every 2 minutes.
I think this will include some timeout spikes on unrelated pages. Applying DB patches to live and busy systems is a learning experience. Yesterday I learned that for some reason I don't yet understand, creating an empty table with foreign key references requires locks on the targeted table that conflicts with normal read/write operations, and that lock can never be fully gained when the table is always being read by at least one transaction. -- Stuart Bishop <stu...@stuartbishop.net> http://www.stuartbishop.net/ _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
