[Wikidata-bugs] [Maniphest] [Updated] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
Lucas_Werkmeister_WMDE added a project: Wikidata-Campsite. TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: Jdforrester-WMF, Mateusz_Konieczny, eranroz, MichaelSchoenitzer_WMDE, Tractopelle-jaune, Florian, Jeff_G, Vachovec1, Tbayer, Ivanhercaz, Lixxx235, Stashbot, Addshore, MrFulano, daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, gerritbot, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Liuxinyu970226, Framawiki, Mainframe98, Stryn, Aklapper, Boshomi, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, aude, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
Addshore added a comment. Relating to where the tmp1 index actually came from T47529#518889TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Jdforrester-WMF, Mateusz_Konieczny, eranroz, MichaelSchoenitzer_WMDE, Tractopelle-jaune, Florian, Jeff_G, Vachovec1, Tbayer, Ivanhercaz, Lixxx235, Stashbot, Addshore, MrFulano, daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, gerritbot, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Liuxinyu970226, Framawiki, Mainframe98, Stryn, Aklapper, Boshomi, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, aude, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
ReleaseTaggerBot edited projects, added MW-1.32-release-notes (WMF-deploy-2018-05-29 (1.32.0-wmf.6)); removed MW-1.32-release-notes (WMF-deploy-2018-05-15 (1.32.0-wmf.4)). TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ReleaseTaggerBotCc: Mateusz_Konieczny, eranroz, MichaelSchoenitzer_WMDE, Tractopelle-jaune, Florian, Jeff_G, Vachovec1, Tbayer, Ivanhercaz, Lixxx235, Stashbot, Addshore, MrFulano, daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, gerritbot, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Liuxinyu970226, Framawiki, Mainframe98, Stryn, Aklapper, Boshomi, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, aude, ArielGlenn, faidon, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, fgiunchedi, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
Addshore added a comment.
The TermSqlIndex::getMatchingTerms method which was patched out during the incident has now been added back.
* 15:11 addshore: Wikibase - Re enable wb_terms things window done
* 14:57 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436007{{!}}TermSqlIndex::getMatchingTerms actually execute select]] (duration: 02m 19s)
* 14:49 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436006{{!}}TermSqlIndex::getMatchingTerms actually execute select]] (duration: 02m 18s)
* 14:32 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436004{{!}}Re add TermSqlIndex::getMatchingTerms select, but dont call]] (duration: 02m 18s)
* 14:29 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436003{{!}}Re add TermSqlIndex::getMatchingTerms select, but dont call]] (duration: 02m 13s)
* 14:13 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436001{{!}}track all wb_terms table access via statsd]] (duration: 02m 19s)
* 14:10 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436000{{!}}track all wb_terms table access via statsd]] (duration: 02m 21s)
ArticlePlaceholder search exposure, PropertySuggester and Special:ItemDisambiguation are still disabled individually.
There is also a new grafana dashboard monitoring the calls to the wb_terms table now. https://grafana-admin.wikimedia.org/dashboard/db/wikibase-wb_terms
I created T195792: Create Mediawiki DB abstraction for individual query timeouts as something that would be useful to have right now for this case to avoid it taking down the whole shard.
Another thing that I would personally like to look into is pages to wikidata deployers for s8 issues / lag and or www.wikidata.org downtime / exceptions (not filed a ticket yet)TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Mateusz_Konieczny, eranroz, MichaelSchoenitzer_WMDE, Tractopelle-jaune, Florian, Jeff_G, Vachovec1, Tbayer, Ivanhercaz, Lixxx235, Stashbot, Addshore, MrFulano, daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, gerritbot, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Liuxinyu970226, Framawiki, Mainframe98, Stryn, Aklapper, Boshomi, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, aude, ArielGlenn, faidon, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
hoo added a project: Wikidata-Ministry-Of-Magic. TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Vachovec1, Tbayer, Ivanhercaz, Lixxx235, Stashbot, Addshore, MrFulano, daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, gerritbot, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Liuxinyu970226, Framawiki, Mainframe98, Stryn, Aklapper, Boshomi, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, aude, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
Vachovec1 added a project: Wikimedia-Incident.Vachovec1 added a comment. Added #wikimedia-incident tag. The incident report alrerady exists...TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Vachovec1Cc: Vachovec1, Tbayer, Ivanhercaz, Lixxx235, Stashbot, Addshore, MrFulano, daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, gerritbot, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Liuxinyu970226, Framawiki, Mainframe98, Stryn, Aklapper, Boshomi, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, aude, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
ReleaseTaggerBot added a project: MW-1.32-release-notes (WMF-deploy-2018-05-15 (1.32.0-wmf.4)). TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ReleaseTaggerBotCc: Stashbot, Addshore, MrFulano, daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, gerritbot, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Liuxinyu970226, Framawiki, Mainframe98, Stryn, Xaosflux, Aklapper, Boshomi, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
daniel added subscribers: Lydia_Pintscher, daniel.daniel added a project: Wikidata.daniel added a comment. Pinging @Lydia_Pintscher. This mostly affected Wikidiata, the site was down for a while (an hour)? As far as I can tell, this is unrelated to the lexeme deployment. It may be related to the wb_terms work @Ladsgroup was doing, and/or something hitting Special:ItemDisambiguation rather hard. Analysis is ongoing. Status summary: PropertySuggester is disabled TermSqlIndex::getMatchingTerms is monkey-patched to return an empty list, https://gerrit.wikimedia.org/r/#/q/I4a8cd1028fd2c5bfc83e8780eea82f0eb79d817f Properties referenced by name from Lua or parser functions are not found (resolvePropertyId returns flase negatives). Special:ItemDisambiguation falsely returns no results ArticlePlaceholder does not offer matches on Special:Search and should probably be disabled for now There are some suggestions of further actions on https://etherpad.wikimedia.org/p/wb_terms_solution. Needs cleaning up and proper task breakdown. Logging more info now, see https://gerrit.wikimedia.org/r/q/Id9fdc74829e6268ecc3861602adfc2eaffc4TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, gerritbot, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Liuxinyu970226, Framawiki, Mainframe98, Stryn, Xaosflux, Aklapper, Boshomi, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
