[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.


  Change 433747 abandoned by Thiemo Kreuz (WMDE):
  Drop search-related indexes when not needed
  
  Reason:
  I'm told it was decided to never alter the wb_terms table again, but replace 
and drop it some day. The linked task got declined. So I assume it's safe to 
abandon this proposal for now. Please restore it if I got that wrong.
  
  https://gerrit.wikimedia.org/r/433747

TASK DETAIL
  https://phabricator.wikimedia.org/T194273

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup, gerritbot
Cc: Addshore, Lydia_Pintscher, Mholloway, Stashbot, Marostegui, 
Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, alaa_wmde, CucyNoiD, 
Nandana, NebulousIris, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, 
Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, 
QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, _jensen, rosalieper, 
Wikidata-bugs, aude, Mbch331, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Ladsgroup]

Ladsgroup added a comment.

In T194273#4793529, @Marostegui wrote:
We cannot alter wb_terms anymore.
 The table is almost 1TB on the master, and the master "only" has 1.1TB free on the master which makes it very high risky and almost impossible to run.


With normalizing of change_tag and dropping of tag_summary it would free up 50-70 GB of space which gives a little bit flexibility but overall I don't see any point of alters since the plan is to just kill the thing.TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LadsgroupCc: Addshore, Lydia_Pintscher, Mholloway, Stashbot, Marostegui, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, CucyNoiD, Nandana, NebulousIris, Banyek, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, _jensen, D3r1ck01, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
We cannot alter wb_terms anymore.
The table is almost 1TB on the master, and the master "only" has 1.1TB free on the master which makes it very high risky and almost impossible to run.TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, MarosteguiCc: Addshore, Lydia_Pintscher, Mholloway, Stashbot, Marostegui, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, CucyNoiD, Nandana, NebulousIris, Banyek, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, _jensen, D3r1ck01, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
Is there any point on having this task if at some point wb_terms is going to be killed?
Is it worth spending the time to research+alter this huge table if it will eventually be killed?TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, MarosteguiCc: Addshore, Lydia_Pintscher, Mholloway, Stashbot, Marostegui, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, CucyNoiD, Nandana, NebulousIris, Banyek, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[jcrespo]

jcrespo added a comment.
I thought that was the query that broke things, am I wrong?TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Lydia_Pintscher, Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[jcrespo]

jcrespo added a comment.
Still ongoing:

Wikibase\Lib\Store\Sql\TermSqlIndex::getMatchingTerms	10.64.32.113	2062	Read timeout is reached (10.64.32.113)	SELECT  term_entity_type,term_type,term_language,term_text,term_weight,term_full_entity_id  FROM `wb_terms`WHERE ((term_language = 'en' AND (term_text LIKE 'ص%' ESCAPE '`' ) AND term_type = 'label' AND term_entity_type = 'item') OR (term_language = 'en' AND (term_text LIKE 'ص%' ESCAPE '`' ) AND term_type = 'alias' AND term_entity_type = 'item'))  LIMIT 2500TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-26T14:25:37Z]  Add tmp1 index back on db1101:3318 - T194273TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435636 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/435636TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435636 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/435636TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-26T09:56:15Z]  Add tmp1 index back on db1087 (sanitarium master), this will generate lag on labsdb hosts - T194273TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435624 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/435624TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435624 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/435624TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-26T05:21:54Z]  Add tmp1 index back on db1099:3318 - T194273TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435492 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3318

https://gerrit.wikimedia.org/r/435492TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435492 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3318

https://gerrit.wikimedia.org/r/435492TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-25T13:15:57Z]  Add indexes back on s8 codfw primary master (db2045) this will generate lag on codfw - T194273TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435128 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/435128TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435128 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/435128TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-25T05:57:24Z]  Add tmp1 index back on dbstore1002 - T194273TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435113 merged by Marostegui:
[operations/puppet@production] mediawiki: Stop replacing term_search_key with empty string

https://gerrit.wikimedia.org/r/435113TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 435113 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] mediawiki: Stop replacing term_search_key with empty string

https://gerrit.wikimedia.org/r/435113TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
Progress of the re-additon of the tmp1  index:


 codfw
 dbstore1002
 labsdb1011
 labsdb1010
 labsdb1009
 db1095 sanitarium
 db1109 main
 db1104 main, api
 db1116 temporary sanitarium
 db1101 rc
 db1099 rc
 db1092 api
 db1087 vslow (master for sanitariums)
 db1071 (master)


We should get the index on db1109, db1104, db1101, db1099, db1092 and db1087 before enabling anything back.
Problably it is not needed on the master, anyways.TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-25T05:05:13Z]  Add tmp1 index back to db1109 - T194273TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Mholloway, Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Ladsgroup]

Ladsgroup added a comment.
Wait a second, I thought it's depooled again. if that's not right, I get to it right now.TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, LadsgroupCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[jcrespo]

jcrespo added a comment.
TermSqlIndex::getMatchingTerms is still failing, it is right now the top failing query among all mediawiki databases:

Wikibase\Lib\Store\Sql\TermSqlIndex::getMatchingTerms	10.64.16.85	2062	Read timeout is reached (10.64.16.85)	SELECT  term_entity_type,term_type,term_language,term_text,term_weight,term_full_entity_id  FROM `wb_terms`WHERE ((term_language = 'en' AND (term_text LIKE 'b%' ESCAPE '`' ) AND term_type IN ('label','alias')  AND term_entity_type = 'property'))  LIMIT 2500TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
I have repooled db1092 and it looks better, but we still have quite some different things and one of the indexes we are removing it is actually being used (term_search_full ):

Look at this:
db1092

root@db1092[wikidatawiki]> show indexes from wb_terms;
+--++--+--+-+---+-+--++--++-+---+
| Table| Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--++--+--+-+---+-+--++--++-+---+
| wb_terms |  0 | PRIMARY  |1 | term_row_id | A |  1671005435 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | wb_terms_text|1 | term_text   | A |   278500905 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | term_full_entity |1 | term_full_entity_id | A |   119357531 | NULL | NULL   | YES  | BTREE  | |   |
+--++--+--+-+---+-+--++--++-+---+
3 rows in set (0.00 sec)

root@db1092[wikidatawiki]> explain SELECT /* Wikibase\Lib\Store\Sql\TermSqlIndex::fetchTerms  */  term_type,term_language,term_text,term_full_entity_id  FROM `wb_terms`WHERE term_entity_type = 'item' AND term_full_entity_id IN (xx)  AND term_language = 'en' AND term_type = 'description';
+--+-+--+---+--+--+-+--+--++
| id   | select_type | table| type  | possible_keys| key  | key_len | ref  | rows | Extra  |
+--+-+--+---+--+--+-+--+--++
|1 | SIMPLE  | wb_terms | range | term_full_entity | term_full_entity | 35  | NULL |  799 | Using index condition; Using where |
+--+-+--+---+--+--+-+--+--++
1 row in set (0.00 sec)

https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3=1=eqiad%20prometheus%2Fops=db1092=9104=1527130879532=1527141679533=30s

db1104:

root@db1104[wikidatawiki]> show indexes from wb_terms;
+--++-+--+-+---+-+--++--++-+---+
| Table| Non_unique | Key_name| Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--++-+--+-+---+-+--++--++-+---+
| wb_terms |  0 | PRIMARY |1 | term_row_id | A |  1702346260 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | wb_terms_entity_id  |1 | term_entity_id  | A |   100138015 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | wb_terms_text   |1 | term_text   | A |   243192322 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | wb_terms_search_key |1 | term_search_key | A |65474856 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | tmp1|1 | term_language   | A | 6375828 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | tmp1|2 | term_type   | A | 8554503 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | tmp1|3 | term_entity_type| A | 6328424 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | tmp1|4 | term_search_key | A |   425586565 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | term_full_entity|1 | term_full_entity_id | A |   113489750 | NULL | NULL   | YES  | BTREE  | |   |
| wb_terms |  1 | term_search_full|1 | term_language   | A | 7063677 | NULL | NULL   | 

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434860 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1092

https://gerrit.wikimedia.org/r/434860TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434860 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1092

https://gerrit.wikimedia.org/r/434860TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
Great, I will repool db1092 tomorrow and check how it goesTASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Ladsgroup]

Ladsgroup added a comment.
Okay, with the patch deployed, I think we are good to go here, @Marostegui : let me know if anything happensTASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, LadsgroupCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-23T17:14:27Z]  Synchronized wmf-config/Wikibase-production.php: SWAT deploying https://gerrit.wikimedia.org/r/#/c/434658/ refs T194273 (duration: 01m 20s)TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434658 merged by jenkins-bot:
[operations/mediawiki-config@master] Make clients read from term_text instead of term_search_key

https://gerrit.wikimedia.org/r/434658TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434658 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] Make clients read from term_text instead of term_search_key

https://gerrit.wikimedia.org/r/434658TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-23T06:49:25Z]  Re-add indexes on wb_terms on db1092 - T194273TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
That query takes 0.00 on db1104 (which has the indexes) and it is taking more than 5 minutes (still running) on db1092 (without the indexes) so I am going to read those indexes to db1092 till this is investigated.TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434632 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/434632TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434632 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/434632TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434631 abandoned by Marostegui:
db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/434631TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
@Ladsgroup I am not sure we should keep on with this task - looks like removing wb_terms_search_key isn't a good idea:
Look at this:

db1092 https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3=1=1527056093345=1527056993345=eqiad%20prometheus%2Fops=db1092=9104

root@db1092[wikidatawiki]> show indexes from wb_terms;
+--++--+--+-+---+-+--++--++-+---+
| Table| Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--++--+--+-+---+-+--++--++-+---+
| wb_terms |  0 | PRIMARY  |1 | term_row_id | A |  1652608049 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | wb_terms_text|1 | term_text   | A |   275434674 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | term_full_entity |1 | term_full_entity_id | A |   118043432 | NULL | NULL   | YES  | BTREE  | |   |
+--++--+--+-+---+-+--++--++-+---+
3 rows in set (0.00 sec)

root@db1092[wikidatawiki]> explain SELECT /* Wikibase\Lib\Store\Sql\TermSqlIndex::getMatchingTerms 2A03:2880:30:2F... */  term_entity_type,term_type,term_language,term_text,term_weight,term_full_entity_id  FROM `wb_terms`WHERE ((term_language = 'bn' AND term_search_key = 'মুসোলিনীর' AND term_type = 'label' AND term_entity_type = 'item') OR (term_language = 'bn' AND term_search_key = 'মুসোলিনীর' AND term_type = 'alias' AND term_entity_type = 'item'))  LIMIT 2500;
+--+-+--+--+---+--+-+--++-+
| id   | select_type | table| type | possible_keys | key  | key_len | ref  | rows   | Extra   |
+--+-+--+--+---+--+-+--++-+
|1 | SIMPLE  | wb_terms | ALL  | NULL  | NULL | NULL| NULL | 1652608451 | Using where |
+--+-+--+--+---+--+-+--++-+
1 row in set (0.00 sec)

db1104: https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3=1=1527056062121=1527056962121=eqiad%20prometheus%2Fops=db1104=9104

root@db1104[wikidatawiki]> show indexes from wb_terms;
+--++-+--+-+---+-+--++--++-+---+
| Table| Non_unique | Key_name| Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--++-+--+-+---+-+--++--++-+---+
| wb_terms |  0 | PRIMARY |1 | term_row_id | A |  1683951612 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | wb_terms_entity_id  |1 | term_entity_id  | A |99055977 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | wb_terms_text   |1 | term_text   | A |   240564516 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | wb_terms_search_key |1 | term_search_key | A |64767369 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | tmp1|1 | term_language   | A | 6378604 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | tmp1|2 | term_type   | A | 8547977 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | tmp1|3 | term_entity_type| A | 6330645 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | tmp1|4 | term_search_key | A |   420987903 | NULL | NULL   |  | BTREE  | |   |
| wb_terms |  1 | term_full_entity|1 | term_full_entity_id | A |   112263440 | NULL | NULL   | YES  | BTREE  | |   |
| wb_terms |  1 | term_search_full|   

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434631 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/434631TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
This is how the table looks like after removing all the indexes that are not needed:

   Table: wb_terms
Create Table: CREATE TABLE `wb_terms` (
  `term_row_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `term_entity_id` int(10) unsigned NOT NULL,
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  `term_entity_type` varbinary(32) NOT NULL,
  `term_language` varbinary(32) NOT NULL,
  `term_type` varbinary(32) NOT NULL,
  `term_text` varbinary(255) NOT NULL,
  `term_search_key` varbinary(255) NOT NULL,
  `term_weight` float unsigned NOT NULL DEFAULT '0',
  PRIMARY KEY (`term_row_id`),
  KEY `wb_terms_text` (`term_text`),
  KEY `term_full_entity` (`term_full_entity_id`)
) ENGINE=InnoDB AUTO_INCREMENT=2259289467 DEFAULT CHARSET=binaryTASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-23T05:43:48Z]  Deploy schema change on db1092 - T191519 T188299 T190148 T194273 T194270TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434624 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/434624TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 434624 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/434624TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, gerritbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
Deletion progress:


 codfw


eqiad:


 dbstore1002
 labsdb1011
 labsdb1010
 labsdb1009
 db1095
 db1109
 db1104
 db1116
 db1101
 db1099
 db1092
 db1087
 db1071
TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-05-23T05:12:05Z]  Deploy schema change on s8 codfw primary master (db2045), this will generate lag on codfw - T194273TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: Stashbot, gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
Thanks for the clarification!
At some point, it would be nice to update the indexes names on Wikibase.sql as on core the index is called wb_terms_text and on the file it is called terms_textTASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[Marostegui]

Marostegui added a comment.
@Ladsgroup can you confirm which indexes we have to drop? There are tw different patches and the syntax doesn't match what we have in production:

These are the patches:

DROP INDEX /*i*/term_search_key ON /*_*/wb_terms;
DROP INDEX /*i*/term_search_full ON /*_*/wb_terms;

DROP INDEX /*i*/term_search ON /*_*/wb_terms;
DROP INDEX /*i*/term_entity ON /*_*/wb_terms;

These are the ones we have in core.

->   KEY `wb_terms_entity_id` (`term_entity_id`),
->   KEY `wb_terms_text` (`term_text`),
->   KEY `wb_terms_search_key` (`term_search_key`),
->   KEY `tmp1` (`term_language`,`term_type`,`term_entity_type`,`term_search_key`),
->   KEY `term_full_entity` (`term_full_entity_id`),
->   KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))

Can you confirm which ones we have to delete as part of this task? (I know we are handling tmp1 on a different task).TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 433747 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Drop search-related indexes on wb_terms when backend of search is different

https://gerrit.wikimedia.org/r/433747TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 433674 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Drop term_entity_id indexes

https://gerrit.wikimedia.org/r/433674TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Minhnv-2809, Maathavan, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T194273: Clean up indexes of wb_terms table

[gerritbot]

gerritbot added a comment.
Change 433674 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Drop term_entity_id indexes

https://gerrit.wikimedia.org/r/433674TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Marostegui, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs