[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
gerritbot added a comment. Change 455630 merged by jenkins-bot: [mediawiki/core@master] Fix undeletion write-both/read-old mode. https://gerrit.wikimedia.org/r/455630TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, gerritbotCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. Thank you anomie for running those queries, which I had suggested or even offered to do to check things look consistent.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
gerritbot added a comment. Change 455630 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler): [mediawiki/core@master] Fix undeletion write-both/read-old mode. https://gerrit.wikimedia.org/r/455630TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, gerritbotCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Anomie added a comment. We should get it fixed before 1.32 is released (or did MCR undeletion make it into 1.31? I forget). For Wikimedia sites I can just make sure everything got migrated as I'm already doing.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, AnomieCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
daniel added a comment. It looks like the "fake" SlotRecord created by RevisionStore::newRevisionFromArchiveRow() passes the $slot->hasRevision() check in RevisionStore::insertRevisionInternal() so it doesn't actually insert it. That's as designed (well, partially): when undeleting, it should not be necessary to insert slot rows, since they should already be there. That's of course not the case when undeleting from an unmigrated archive row. Nasty edge case, lucky we still write the "old" info the revision table, otherwise we would have lost these revisions! This will never happen when the database in a consistently migrated or unmigrated state, but we probably want to properly deal with this in RevisionStore. How urgent do you think this is?TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, danielCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Anomie added a comment. Also it looks like undeletions can somehow manage to escape the script, if the undeletion happens between when the rev_id is reached in revision and when the ar_rev_id is reached in archive. Found a few on commonswiki so far, and I won't be too surprised if there are more on enwiki or other big wikis. It looks like the "fake" SlotRecord created by RevisionStore::newRevisionFromArchiveRow() passes the $slot->hasRevision() check in RevisionStore::insertRevisionInternal() so it doesn't actually insert it.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, AnomieCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-27T16:31:53Z] Running populateContentTables.php on advisorswiki for T183488 and T202904TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, StashbotCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
daniel added a comment. In T183488#4535083, @Anomie wrote: I'm running queries on all wikis to see if there are any revision or archive rows lacking a slots row. That found T202904 so far. Wow, nice find!TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, danielCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
daniel added a comment. @Anomie What do you think of re-running the script on a few more wikis, as a spot check to verify that it finds nothing to do? If otoh the script finds stuff to migrate, something is wrong...TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, danielCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-27T15:49:02Z] Re-running populateContentTables.php on aawikibooks, gotwikibooks, kswikiquote, lvwikibooks, nostalgiawiki, wawikibooks and wikimania2005wiki for T183488TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, StashbotCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-22T19:01:36Z] re-running populateContentTables.php on cawiki for T183488TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, StashbotCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
tstarling added a comment. enwiki is complete now, so only the T202032 wikis remain.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarlingCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Anomie added a comment. In T183488#4504507, @tstarling wrote: So how do we end up trying to insert a row for revision 3003 twice? max(rev_id) on this wiki is 3002, so the ar_rev_id certainly came from populateArchiveRevId.php, which allocated ar_rev_id values by inserting dummy rows into revision and deleting them in the same transaction. Or deduplicateArchiveRevId.php, which does the same thing. It may be that MySQL has some method to reallocate autoincremented IDs in this case which we didn't know about. wikiadmin@10.64.0.205(aawikibooks)> select max(ar_rev_id) from archive; ++ | max(ar_rev_id) | ++ | 3275 | ++ wikiadmin@10.64.0.205(aawikibooks)> SELECT auto_increment FROM information_schema.tables WHERE table_schema = 'aawikibooks' and table_name = 'revision'; ++ | auto_increment | ++ | 3152 | ++ That's certainly not right. https://dev.mysql.com/doc/refman/8.0/en/innodb-auto-increment-handling.html#innodb-auto-increment-initialization may be relevant here: MySQL until 8.0 (and MariaDB until some version after 10.1.29, apparently, based on local testing) doesn't actually save the auto-increment value to disk. Instead on server restart it recalculates the value via the equivalent of SELECT MAX(...). If MAX(rev_id) < MAX(ar_rev_id) on a restart, we'll likely wind up with reused revision IDs for new revisions. Once MCR is rolled out and the slots table is populated, and as long as every revision has a main slot, future instances of this problem on restart would cause the next MAX(ar_rev_id) - MAX(rev_id) revision inserts to fail thanks to the slots table PK. Each failure would increment the rev_id auto-increment counter, so the problem would eventually go away until the next master restart that happens to have problematic state. We might be able to have the same sort of thing on a master failover, if the new master's rev_id auto-increment counter is somehow out of sync in a similar manner.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, AnomieCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
daniel added a comment. Duplicates are prevented by the unique index on slot_revision_id/slot_role_id. Orphan slot rows are possible, just as orphan revision rows have been possible before. We can run a query to detect those on some wikis as a spot check. If we find nothing, I don't think full validation is needed. You are right that all code should have tests - so ideally, the migration script should have a unit test that validates the database content. We opted for manual testing with "real" data instead, since this is a one-off conversion, and "fake" data is unlikely to expose problems anyway.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, danielCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. So the script will make sure to not miss any content, but doesn't assure it introduces extra one, or duplicates some, that is why I mention the usage of extra checks. Takes very little time (1 long running query per check) and validates the whole process. The same thing that "all code should have tests" "all tasks should have a way to validate they completed correctly". I think this is very process to miss that, don't you think?TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
daniel added a comment. @jcrespo Ooops, I updated my comment after you already answered, sorry.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, danielCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. In T183488#4506824, @daniel wrote: We did pretty extensive testing of this on the test copies (db and db1112). We could have a script that does basic sanity checks, but I'm not sure what to test, beyond the trivial. All I can think of is: all revision rows have an associated slot row. all archive rows have an associated slot row. all slot rows have an associated content. ar_rev_id as no conflicts with rev_id ar_rev_id as no conflicts with itself Is that what you had in mind? Yes, also the inverse checks, assuming it is 1:1 at the moment.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
daniel added a comment. it would be nice to run a read-only process to check the consistency of the migration We did pretty extensive testing of this on the test copies (db and db1112). We could have a script that does basic sanity checks, but I'm not sure what to test, beyond the trivial. All I can think of is: all revision rows have an associated slot row. all archive rows have an associated slot row. all slot rows have an associated content. ar_rev_id as no conflicts with rev_id ar_rev_id as no conflicts with itself Is that what you had in mind?TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, danielCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. I mentioned this to Tim, and he said to better mention it to Brad (and Daniel?), that it would be nice to run a read-only process to check the consistency of the migration- it should take much less time than the writes and would avoid headaches given the issues with the existing data and other possible complications that could have happened during the migration. I don't think anything is problem-free, but we can try to detect the issues as much as we can. What do you think?TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
tstarling added a comment. So how do we end up trying to insert a row for revision 3003 twice? max(rev_id) on this wiki is 3002, so the ar_rev_id certainly came from populateArchiveRevId.php, which allocated ar_rev_id values by inserting dummy rows into revision and deleting them in the same transaction. It may be that MySQL has some method to reallocate autoincremented IDs in this case which we didn't know about.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarlingCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-14T11:40:41Z] restarted populateContentTables.php on s2 (T183488)TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, StashbotCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
tstarling added a comment. You can see the full logs at mwmaint1001:/var/log/mediawiki/populateContentTables/ . On both aawikibooks and gotwikibooks, the error occurred on the second batch of the archive table, starting at ar_rev_id 2001. In both cases it was also the last batch, with the maximum ar_rev_id being 3275 and 3175 respectively.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarlingCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
daniel added a comment. aawikibooks failed with "Error: 1062 Duplicate entry '3003-1' for key 'PRIMARY' (10.64.0.205)" It's annoying that this error message doesn't say which *table* this was on. My guess is that it'S the slots table, which has primary key ( slot_revision_id, slot_role_id ). slot_role_id is always 1 for now (for the main slot). So how do we end up trying to insert a row for revision 3003 twice? My first guess was that we have the same revision ID in the revision and the archive table, from an incomplete deletion. This is however not the case. Instead, there is two rows in the archive table with the same ar_rev_id, for what seems to be completely different revisions: MariaDB [aawikibooks_p]> select ar_id, ar_rev_id, ar_page_id, ar_namespace, ar_title, ar_timestamp, ar_text_id from archive where ar_rev_id = 3003\G *** 1. row *** ar_id: 29 ar_rev_id: 3003 ar_page_id: NULL ar_namespace: 8 ar_title: Accesskey-watch ar_timestamp: 20060701183713 ar_text_id: 2956 *** 2. row *** ar_id: 987 ar_rev_id: 3003 ar_page_id: NULL ar_namespace: 8 ar_title: Group-sysop ar_timestamp: 20060701183714 ar_text_id: 3003 2 rows in set (0.00 sec) That's a bit worrying. How did that happen? Anyway, one of them needs to get a fresh revision id, probably best by setting ar_rev_id to NULL, and then running populateArchiveRevId.php to make sure the new revision ID is assigned safely.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, danielCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
tstarling added a comment. Log summary: aawikibooks failed with "Error: 1062 Duplicate entry '3003-1' for key 'PRIMARY' (10.64.0.205)" cawiki failed with "Replication wait failed: Server shutdown in progress" and will need to be restarted. gotwikibooks also failed with a duplicate key error s5 (dewiki) is complete s2 was killed So 4 concurrent processes are currently running: s1, s3, s6 and s7.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarlingCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
tstarling added a comment. In T183488#4499288, @jcrespo wrote: @tstarling Please stop writes going to *s2* unless they have already finished Done. s2 was up to itwiki rev_id 3012040. enwiki should take about another 6 days at the current rate.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarlingCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
jcrespo added a comment. Let's keep an eye on performance metrics, specially https://grafana.wikimedia.org/dashboard/db/save-timing Although with the estimations given T183488#4480579, it may be desirable to make things faster rather and short than extended for a long period of time. Meanwhile, I will keep an eye on the disk usage and size.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: jcrespo, greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-13T01:45:10Z] on mwmaint1001 running populateContentTables.php on all wikis using ~tstarling/pct-list T183488TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, StashbotCc: greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-07T03:46:00Z] on mwmaint1001 running populateContentTables.php concurrently on wikidatawiki and commonswiki (T183488)TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, StashbotCc: greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Fjalapeno, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
greg added a comment. ack, thanks!TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, gregCc: greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
tstarling added a comment. @greg The WN31 things are done now, only 1081 seconds for mediawikiwiki and 9252 seconds for metawiki. For metawiki the rate was about the same as anomie got for testwiki, 2000 rows per second for the revision table and 600 rows per second for the archive table. At that rate, we can expect wikidatawiki to take about 91 hours and commonswiki to take about 48 hours. We can run them concurrently since they are on different DB clusters, and that way maybe get them done by the end of the week.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, tstarlingCc: greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-06T04:29:45Z] on mwmaint1001 running populateContentTables.php on metawiki T183488TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, StashbotCc: greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-06T04:09:55Z] on mwmaint1001 running populateContentTables.php on mediawikiwiki T183488TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, StashbotCc: greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
greg added a comment. In T183488#4471137, @tstarling wrote: Daniel proposed the following schedule: Today: test2wiki, testwikidatawiki This week (WN31): mediawikiwiki, metawiki Next week (WN32): wikidatawiki, commonswiki Week after (WN33): everything Given it's Friday I don't think the rest of this week (WN31) will happen :) Should this schedule be updated? Please do add the correct wikis runs to the "week of" section of the deployment calendar in the corresponding week: https://wikitech.wikimedia.org/wiki/DeploymentsTASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, gregCc: greg, tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
tstarling added a comment. test2wiki and testwikidatawiki are complete.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, tstarlingCc: tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-08-02T05:30:43Z] on mwmaint1001 running populateContentTables.php as described in T183488TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, StashbotCc: tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
tstarling added a comment. Daniel proposed the following schedule: Today: test2wiki, testwikidatawiki This week (WN31): mediawikiwiki, metawiki Next week (WN32): wikidatawiki, commonswiki Week after (WN33): everything I'm going to start on this now.TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, tstarlingCc: tstarling, Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-07-25T15:31:45Z] running populateContentTables.php on testwiki for T183488TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, StashbotCc: Stashbot, Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Ltrlg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
gerritbot added a comment. Change 406595 merged by jenkins-bot: [mediawiki/core@master] [MCR] RevisionStore, enable insertions for new schema https://gerrit.wikimedia.org/r/406595TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, gerritbotCc: Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, JJMC89, Maathavan, Agabi10, Susannaanas, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Abit added a comment. In T183488#4027864, @Addshore wrote: In T183488#4027858, @Abit wrote: @Addshore, the MCR pert says you and Katie are working on this, and I see you wrote a related gerrit patch. Should this be assigned to you? So I am writing some of the underlying code, but that is probably best tracked in T174024. I won't be doing the actual migration, but I believe is what this ticket is target towards. Good to know, thank you!TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AbitCc: Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, SandraF_WMF, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, Maathavan, Susannaanas, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
Abit added a comment. @Addshore, the MCR pert says you and Katie are working on this, and I see you wrote a related gerrit patch. Should this be assigned to you?TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AbitCc: Abit, gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Giuliamocci, Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Baloch007, E1presidente, Ramsey-WMF, Cparle, Darkminds3113, SandraF_WMF, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, Lewizho99, Maathavan, Susannaanas, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183488: MCR schema migration stage 2: populate new fields
gerritbot added a comment. Change 406595 had a related patch set uploaded (by Addshore; owner: Addshore): [mediawiki/core@master] [MCR] RevisionStore, enable insertions for new schema https://gerrit.wikimedia.org/r/406595TASK DETAILhttps://phabricator.wikimedia.org/T183488EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Jdforrester-WMF, Anomie, Addshore, aude, Aklapper, daniel, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Susannaanas, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
