[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 physikerwelt phy...@ckurs.de changed: What|Removed |Added CC||phy...@ckurs.de --- Comment #26 from physikerwelt phy...@ckurs.de --- Is there an option to skip this update in update.php? I tried mwscript update.php --quick --nopurge --skip-compat-checks, to run updated that follow the step Updating *_from_namespace fields in links tables. But I did nothing helped. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 --- Comment #25 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 148296 abandoned by Brian Wolff: Reduce batch size of populateBacklinkNamespace from 200 to 20 Reason: gerrit change 151027 addresses the same issue but probably more robustly https://gerrit.wikimedia.org/r/148296 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 Kevin Israel (PleaseStand) pleasest...@live.com changed: What|Removed |Added CC||pleasest...@live.com --- Comment #24 from Kevin Israel (PleaseStand) pleasest...@live.com --- (In reply to Greg Grossmeier from comment #22) It already happened in production. Which is the only reason why it was merged to begin with. I'm not sure, but in production, this update may still be in progress. The only entry in https://wikitech.wikimedia.org/wiki/Server_Admin_Log I see regarding this is under July 30: 21:04 AaronSchulz: Started populateBacklinkNamespace.php on wikidata and commons. The schema change done before the change was merged was to add some new columns and set them to a default value of 0. The update referred to in this report (populateBacklinkNamespace script) would happen afterward, setting the correct values for those columns. That's why a $wgUseLinkNamespaceDBFields setting was added. It is not currently enabled in production. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 --- Comment #19 from Antoine hashar Musso has...@free.fr --- I am also wondering how we are going to handle that update in production. Might end up taking a long time as well. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 --- Comment #20 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to Antoine hashar Musso from comment #19) I am also wondering how we are going to handle that update in production. Might end up taking a long time as well. Towards the end, beta was updating about 200 rows every 30 seconds. enwiki's page_ids go up to 43371588, which gives ((30/200)*43371588)/(60*60*24) = 75.2 So 75 days to update enwiki (assuming similar performance, which is questionable. enwiki has much more powerful db so can probably do the update faster. OTOH, it should probably have a much smaller batchsize, which could potentially slow down the update. So who knows). Anyways, taking that very rough guess at face value, if the update takes 2.5 months, I don't see any problem. There's no deadline for when the update has to finish by. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 --- Comment #21 from Tim Landscheidt t...@tim-landscheidt.de --- (In reply to Bawolff (Brian Wolff) from comment #20) (In reply to Antoine hashar Musso from comment #19) I am also wondering how we are going to handle that update in production. Might end up taking a long time as well. Towards the end, beta was updating about 200 rows every 30 seconds. enwiki's page_ids go up to 43371588, which gives ((30/200)*43371588)/(60*60*24) = 75.2 So 75 days to update enwiki (assuming similar performance, which is questionable. enwiki has much more powerful db so can probably do the update faster. OTOH, it should probably have a much smaller batchsize, which could potentially slow down the update. So who knows). Anyways, taking that very rough guess at face value, if the update takes 2.5 months, I don't see any problem. There's no deadline for when the update has to finish by. I'm no DBA, but running three UPDATEs for every page row doesn't sound like the brightest idea. I'm pretty sure MariaDB has much nicer performance if you speak to it in SQL like you proposed in comment #9. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 --- Comment #22 from Greg Grossmeier g...@wikimedia.org --- (In reply to Antoine hashar Musso from comment #19) I am also wondering how we are going to handle that update in production. Might end up taking a long time as well. It already happened in production. Which is the only reason why it was merged to begin with. Remember folks: If your code goes to production and you want to make a database change, file a Schema Change bug and have our DBA (Sean) take care of it BEFORE you merge. Aaron did that right. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 Antoine hashar Musso has...@free.fr changed: What|Removed |Added Status|REOPENED|RESOLVED Resolution|--- |FIXED --- Comment #23 from Antoine hashar Musso has...@free.fr --- Excellent! So there is nothing to talk about anymore =) Beta is happy, slave lag is back to 0 seconds. Topic closed. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 Chris McMahon cmcma...@wikimedia.org changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=65486 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 Antoine hashar Musso has...@free.fr changed: What|Removed |Added CC||zfili...@wikimedia.org --- Comment #12 from Antoine hashar Musso has...@free.fr --- *** Bug 68373 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 --- Comment #13 from Antoine hashar Musso has...@free.fr --- Created attachment 15999 -- https://bugzilla.wikimedia.org/attachment.cgi?id=15999action=edit log of update.php for the beta cluster simplewiki Aaron might be interested. On beta simple wiki (which has roughly 250k pages), the console run is https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=simplewiki/2742/console (attached to bug report) -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 Antoine hashar Musso has...@free.fr changed: What|Removed |Added CC||aschulz4...@gmail.com --- Comment #14 from Antoine hashar Musso has...@free.fr --- Aaron Schulz might be interested in this bug report. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 --- Comment #15 from Bawolff (Brian Wolff) bawolff...@gmail.com --- FWIW, the update.php job finished successfully. lag on deployment-db2 seems to be holding at about 3 hours and 20 minutes for now. Things will probably be back to normal in several hours. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 Bawolff (Brian Wolff) bawolff...@gmail.com changed: What|Removed |Added Status|PATCH_TO_REVIEW |RESOLVED Resolution|--- |FIXED --- Comment #16 from Bawolff (Brian Wolff) bawolff...@gmail.com --- Slave lag is back down to 0. Guess this is fixed. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 Greg Grossmeier g...@wikimedia.org changed: What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED |--- --- Comment #17 from Greg Grossmeier g...@wikimedia.org --- I want to leave this open until we've figured out if we can prevent this from happening again. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 --- Comment #18 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to Greg Grossmeier from comment #17) I want to leave this open until we've figured out if we can prevent this from happening again. Well update is done. Update only gets run once so it wont happen again on beta wiki unless someone manually runs populateBacklinkNamespace.php --force, or deletes the relavent entry in the updatelog table. The deeper issue of course is the population script had too big a batch size. If you want to see if that is fixed i guess it might make sense to remove the line from updatelog on beta after merging comment 10 to see if the update still explodes. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 68349] populateBacklinkNamespace script causing massive slave lag on beta
https://bugzilla.wikimedia.org/show_bug.cgi?id=68349 Bawolff (Brian Wolff) bawolff...@gmail.com changed: What|Removed |Added Summary|The current db schema |populateBacklinkNamespace |change upgrade is taking|script causing massive |far too long|slave lag on beta -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
