[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
daniel added a comment. The above patch is an alternative attempt to fix the lock retention issue. It's the best I can think of. Not sure this will fix the problem, though.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
gerritbot added a comment. Change 442883 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler): [mediawiki/core@wmf/1.32.0-wmf.10] Minimize the work done within atomic section in insertRevisionon(). https://gerrit.wikimedia.org/r/442883TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
gerritbot added a comment. Change 442882 abandoned by Daniel Kinzler: Minimize the work done within atomic section in insertRevisionon(). Reason: should not be on master https://gerrit.wikimedia.org/r/442882TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
gerritbot added a comment. Change 442882 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler): [mediawiki/core@master] Minimize the work done within atomic section in insertRevisionon(). https://gerrit.wikimedia.org/r/442882TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
daniel added a comment. Here are a few things I poked at, without finding anything relevant: @Tgr suspects that the something is grabbing a FOR UPDATE lock on revision_comment_temp. But the only code that seems to do that seems to be in WikiPage::doDeleteArticleReal(), which shouldn't be called so often that it would cause trouble. Another suspicion was that some code that calls getQueryInfo on Revision or RevisionStore logs the tables returned by that method. However, getQueryInfo() has always returned revision_comment_temp, and I found no new usages of getQueryInfo in the diff between wmf8 and wmf10. Also it seems to me that in wmf8, revision_comment_temp was already being updated while holding a FOR UPDATE lock on the page table, at least during edits: WikiPage::doModify calls startAtomic() and lockAndGetLatest(), grabbing a FOR UPDATE lock on the page table. Then it calls Revision::insertOn which calls RevisionStore->insertRevisionOn which calls CommentStore->insertWithTempTable which modifies revision_comment_temp.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, danielCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
daniel added a comment. After staring at the code a bit, my best guess is: The MCR refactoring introduced doAtomicSection() to RevisionStore::insertRevisionOn(), to preserve consistency between the revision, slots, and content tables. The atomic section also includes the code for writing the blobs, and the code for inserting into the comment table. Previously, insertRevisionOn() did not explicitly start any kind of transaction, though in the context of WikiPage::doEditContent(), it has always been used after startAtomic(), so all of this was already covered by a transaction. But this was not necessarily the case when insertRevisionOn() resp Revision::insertOn() was called directly, e.g. when inserting a null revision during file upload.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, danielCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Tgr added a comment. There are about 1000 errors (in the deploy window, on Commons/Wikidata) where the failing query is on the revision_comment_temp table, and only about 50 where it is not. So I think it is fair to assume that is the primary cause.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, TgrCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
daniel added a comment.
The DBPerformance log shows a spike during the time wmf-10 was deployed on group1: https://logstash.wikimedia.org/goto/7c86a7d63a305c220a37a3a49844ef2c. The vast majority of entries are for commonswiki. Here are a few examples:
Sub-optimal transaction on DB(s) [10.64.48.23 (commonswiki) (TRX#26770d)]:
0 0.000436 query-m: INSERT IGNORE INTO `page` (page_namespace,page_title,page_restrictions,page_is_redirect,page_is_new,page_random,page_touched,page_latest,page_len) VALUES ('X') [TRX#26770d]
1 0.001476 query-m: INSERT INTO `blobs_cluster2N` (blob_text) VALUES ('X')
2 0.000313 query-m: INSERT INTO `text` (old_id,old_text,old_flags) VALUES (NULL,'X') [TRX#26770d]
3 0.000497 query-m: INSERT INTO `comment` (comment_hash,comment_text,comment_data) VALUES ('X',NULL) [TRX#26770d]
4 0.000425 query-m: INSERT INTO `revision` (rev_page,rev_parent_id,rev_text_id,rev_minor_edit,rev_timestamp,rev_deleted,rev_len,rev_sha1,rev_comment,rev_user,rev_user_text,rev_content_model,rev_content_format) VALUES ('X',NULL,NULL) [TRX#26770d]
5 15.847574 query-m: INSERT INTO `revision_comment_temp` (revcomment_rev,revcomment_comment_id) VALUES ('X') [TRX#26770d]
This is from https://commons.wikimedia.org/w/index.php?title=File:Portrait_of_Maria_van_Rijswijk_Dutch_School_Rijksdienst_voor_het_Cultureel_Erfgoed_B670.jpg&action="">
I'm surprised to see blobs_cluster2N in there - ExternalStore's blob tables are generally on a different DB server, no? I'll check whether this may be the cause of the problem somehow.
Sub-optimal transaction on DB(s) [10.64.48.23 (commonswiki) (TRX#e7bddd)]:
0 0.000614 query-m: UPDATE `image` SET img_name = 'X' [TRX#e7bddd]
1 0.000412 query-m: UPDATE `image_comment_temp` SET imgcomment_name = 'X' [TRX#e7bddd]
2 0.000372 query-m: UPDATE `oldimage` SET oi_name = 'X' [TRX#e7bddd]
3 0.000363 query-m: INSERT INTO `comment` (comment_hash,comment_text,comment_data) VALUES ('X',NULL) [TRX#e7bddd]
4 0.000396 query-m: INSERT INTO `revision` (rev_page,rev_parent_id,rev_text_id,rev_minor_edit,rev_timestamp,rev_deleted,rev_len,rev_sha1,rev_comment,rev_user,rev_user_text,rev_content_model,rev_content_format) VALUES ('X',NULL,NULL) [TRX#e7bddd]
5 15.724425 query-m: INSERT INTO `revision_comment_temp` (revcomment_rev,revcomment_comment_id) VALUES ('X') [TRX#e7bddd]
This was a POST to /w/api.php. Presumably an upload.
Sub-optimal transaction on DB(s) [10.64.48.23 (commonswiki) (TRX#06688b)]:
0 0.031600 query-m: INSERT INTO `text` (old_id,old_text,old_flags) VALUES (NULL,'X') [TRX#06688b]
1 0.000452 query-m: INSERT INTO `revision` (rev_page,rev_parent_id,rev_text_id,rev_minor_edit,rev_timestamp,rev_deleted,rev_len,rev_sha1,rev_comment,rev_user,rev_user_text,rev_content_model,rev_content_format) VALUES ('X',NULL,NULL) [TRX#06688b]
2 16.074089 query-m: INSERT INTO `revision_comment_temp` (revcomment_rev,revcomment_comment_id) VALUES ('X') [TRX#06688b]
From TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, danielCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Tgr added a comment. In T198350#4321620, @matmarex wrote: The errors also did not appear (or at least not in notable numbers) when the changes were deployed to the first set of production wikis on Tuesday (mediawiki.org and test.wp, test2.wp). If this is really MCR-related, that was on the testwikis for two weeks as wmf.999. There is not a single error from that branch. (Then again there is not a single error from any wiki other than Commons / Wikidata, so all that says is that group0 is not really useful for catching load-related problems.)TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, TgrCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
daniel added a comment. @Aklapper As far as I can see, recent instance of the first two issues T179884 and T197464#4321254, were probably caused by this. The "overwriting image" one (T198177) seems off. It may still be a consequence somehow, but that issue is about updates to the image table, not about page revisions.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, danielCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
daniel added a comment. In T198350#4321620, @matmarex wrote: In T198350#4320899, @AlexisJazz wrote: Assuming this was not a case of "It compiles, ship it!" I am curious as to why this wasn't noticed when testing. It appears to require multiple users making actions on a wiki simultaneously, and usually when you test a change you're the only user doing anything on your local testing wiki. Not just some concurrent users. Massive concurrent load. It's a performance problem. These are pretty much impossible to test for. We don't have a duplicate of our server infrastructure under full emulated load.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, danielCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
daniel added a comment. I created a patch that reverts the MCR patches related to RevisionStore, but keeps the change that introduces PageUpdater. We could deploy the branch with the RevisionStore stuff reverted, and see if it still blows up. Whether or not it does, we'll know more about the cause. Bisecting by potentially bringing the site down is not a great option, but I honestly don't have a better idea. Unless perhaps we had some way to generate sufficient load on a test system to trigger the error.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, danielCc: WMDE-leszek, Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
gerritbot added a comment. Change 442834 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler): [mediawiki/core@wmf/1.32.0-wmf.10] Revert MCR RevisionStore changes https://gerrit.wikimedia.org/r/442834TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, gerritbotCc: Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
matmarex added a comment. In T198350#4320899, @AlexisJazz wrote: Assuming this was not a case of "It compiles, ship it!" I am curious as to why this wasn't noticed when testing. It appears to require multiple users making actions on a wiki simultaneously, and usually when you test a change you're the only user doing anything on your local testing wiki. The errors also did not appear (or at least not in notable numbers) when the changes were deployed to the first set of production wikis on Tuesday (mediawiki.org and test.wp, test2.wp).TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, matmarexCc: Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
jcrespo added a comment.
{icon heart} Tgr analysisTASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, jcrespoCc: Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
DerHexer added a comment. I got the first error when I used Special:Nuke on Wikimedia Commons. Is it possible that something has not been updated with this old tool?TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, DerHexerCc: Danmichaelo, jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
jcrespo added a comment. @Addshore Most of those you point happen all the time, unlike the ones @Marostegui pointed, which are new.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, jcrespoCc: jcrespo, Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Addshore added a comment.
In T198350#4320941, @Marostegui wrote:
From what I can see it was not only related to that table and to that write, there are lots of others, but the INSERT INTO revision_comment_temp (revcomment_rev,revcomment_comment_id) VALUES ('xx','x') appears quite a lot. There are around 1000 errors for that table on that specific timeframe: https://logstash.wikimedia.org/goto/43b11908e6569dd05fc09fe450c9ed06
There are even more other lock related errors during the period:
Could not acquire lock 'CategoryMembershipUpdates:3673061'
Function: Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages
Error: 1213 Deadlock found when trying to get lock; try restarting transaction
WikiPage::insertOn 10.64.48.23 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.48.23) INSERT IGNORE INTO `page` (page_namespace,page_title,page_restrictions,page_is_redirect,page_is_new,page_random,page_touched,page_latest,page_len) VALUES ('6','The_Parkville_in_May_2018.jpg','','0','1','0.604298787837','20180627195904','0','0')
And some timeouts:
IndexPager::buildQueryInfo (LogPager) 10.64.0.164:3314 2062 Read timeout is reached (10.64.0.164:3314) SELECT log_id,log_type,log_action,log_timestamp,log_namespace,log_title,log_params,log_deleted,user_id,user_name,user_editcount,COALESCE( comment_log_comment.comment_text, log_comment ) AS `log_comment_text`,comment_log_comment.comment_data AS `log_comment_data`,comment_log_comment.comment_id AS `log_comment_cid`,log_user,log_user_text,NULL AS `log_actor`,(SELECT GROUP_CONCAT(ct_tag SEPARATOR ',') FROM `change_tag`WHERE ct_log_id=log_id ) AS `ts_tags` FROM `logging` LEFT JOIN `comment` `comment_log_comment` ON ((comment_log_comment.comment_id = log_comment_id)) LEFT JOIN `user` ON ((user_id=log_user)) WHERE (log_type NOT IN ('spamblacklist','titleblacklist','abusefilterprivatedetails','suppress')) AND ((log_user = '654')) AND ((log_deleted & 4) = 0) AND (log_type != 'thanks') AND (log_type != 'patrol') AND (log_type != 'tag') ORDER BY log_timestamp DESC LIMIT 51
TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, AddshoreCc: Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Tgr added a comment. There has always been a slow but steady stream of lock timeouts: https://logstash.wikimedia.org/app/kibana#/dashboard/default?_g=h@e2dcd68&_a=h@5f113f2 F23048550: logstash.wikimedia.org_app_kibana.png revision_comment_temp is the only one that spiked after the deploy(*), so presumably the others are unrelated. (*) Probably. Same graph with revision_comment_temp filtered out: F23048567: logstash.wikimedia.org_app_kibana (1).png Our logging could be better, though.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, TgrCc: Ankry, Nikerabbit, Marostegui, Anomie, cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
AlexisJazz added a comment. Assuming this was not a case of "It compiles, ship it!" I am curious as to why this wasn't noticed when testing.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, AlexisJazzCc: cscott, daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Yann added a comment. There are at least 150 files without a page (and probably more). Starting from https://commons.wikimedia.org/wiki/File:NIG-ARG_(1).jpg at 21:40 (at least) until https://commons.wikimedia.org/wiki/File:Fifteen_pen_and_ink_maps_of_the_battle_of_the_Wilderness_LOC_2007627346-11.jpg at 21:59TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, YannCc: daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Tgr added a comment. 1770 errors in that one hour, only 240 of them have Wikibase in the stack trace, so not really Wikidata related. They do all target revision_comment_temp. Would be nice to know where the other leg of that lock is. In theory there should only be a conflict when two processes try to add a comment for the same revision, which should not be possible.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, TgrCc: daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
daniel added a comment. Is that stack trace representative? is it always INSERT INTO revision_comment_temp? It's quite possible that this was caused by the MCR patches, but so far, I don't have any clue as to how or why.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, danielCc: daniel, Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Yann added a comment. Yes, there are a number of files without pages around 22:00 UTC https://commons.wikimedia.org/w/index.php?title=Special:NewFiles&offset=20180627200112&limit=50TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, YannCc: Tgr, Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Raymond added a comment. I guess a cleanup routine/check is necessary now. Half-done image uploads like https://commons.wikimedia.org/wiki/File:Typhoon_MyGuide_3500_mobile_-_controller_-_Intel_PXA255A0C300-1180.jpg (missing file page). For this one I will try to create the file page manually.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, RaymondCc: Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Addshore added a comment. Looking at logstash this should now have recoveredTASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, AddshoreCc: Addshore, Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Herzi.Pinki added a comment. for me it's 100% failuresTASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, Herzi.PinkiCc: Herzi.Pinki, DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Yann added a comment. Again [WzPsbApAAD0AAFX1EFwAAABX] 2018-06-27 19:58:52: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, YannCc: DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Yann added a comment. Again with the same file: [WzPrpApAIEIAAJ1@bdMAAACA] 2018-06-27 19:55:32: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, YannCc: DC, Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Yann added a comment. [WzPq4gpAIDAAAH@yqFsW] 2018-06-27 19:52:19: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" while trying to edit https://commons.wikimedia.org/wiki/File:SHKF-logo.pngTASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, YannCc: Yann, Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-06-27T19:52:12Z] Rolling back group1 due to rise in error rate (T198350)TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, StashbotCc: Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
dduvall added a comment. A number of SlowTimer errors have shown up in fatalmonitor as well. I'm going to roll back the train for now.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvallCc: Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
DerHexer added a comment. Throws API errors: API request failed (internal_api_error_DBQueryError): [WzPl3wpAIDYAAF1ZcmYC] Database query error. at Wed, 27 Jun 2018 19:30:57 GMT served by mw1342 And UI errors: [WzPmNgpAMFQAAK-3yloW] 2018-06-27 19:32:23: Fataler Ausnahmefehler des Typs „Wikimedia\Rdbms\DBQueryError“ Leaves some images on Commons broken: https://commons.wikimedia.org/wiki/File:DJM_und_SMK_2018-05-31_WK_621_200m_Schmetterling_weiblich_Mehrkampf_013.jpgTASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, DerHexerCc: Raymond, DerHexer, matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment
AlexisJazz added a comment. Just uploaded a screenshot, cropped it a bit, tried to upload a new version over it, guess what? Database error A database query error has occurred. This may indicate a bug in the software. [WzPlqwpAIDAAAENRhT8AAADP] 2018-06-27 19:30:05: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" The crop WAS uploaded! But there is only one revision. Reported dimensions: 808 pixels wide. Actual image: 800 pixels.TASK DETAILhttps://phabricator.wikimedia.org/T198350EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dduvall, AlexisJazzCc: matmarex, AlexisJazz, Aklapper, greg, Jdforrester-WMF, Stashbot, gerritbot, MarcoAurelio, revi, dduvall, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
