[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

2020-10-07 Thread Marostegui
Marostegui added a comment. The only common query I have found that executes on all hosts is: SELECT /* Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin 141.136.76.164 */ wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbpt_property_id FROM

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

2020-10-07 Thread Marostegui
Marostegui added a comment. I forgot to paste: root@mwmaint2001:~# crontab -l -uwww-data | grep -w wikidatawiki */3 * * * * echo "$$: Starting dispatcher" >> /var/log/wikidata/dispatchChanges-wikidatawiki.log; /usr/local/bin/mwscript extensions/Wikibase/

[Wikidata-bugs] [Maniphest] T245818: Audit query pages on Wikidata

2020-10-06 Thread Marostegui
Marostegui added a comment. Sorry @Ladsgroup - totally missed this! So far I haven't seen any other query or cronjob that would consistently make us going and kill it. Thank you! TASK DETAIL https://phabricator.wikimedia.org/T245818 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

2020-10-06 Thread Marostegui
Marostegui added a project: Wikidata. Marostegui added subscribers: Ladsgroup, Addshore. Marostegui moved this task from Triage to In progress on the DBA board. Marostegui added a comment. Adding @Addshore and @Ladsgroup as they have lots of cool dashboards (that I am unable to find) where

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-27 Thread Marostegui
Marostegui closed this task as "Resolved". Marostegui added a comment. @Ladsgroup I am going to close this - once you've got all the changes merged, let's create a normal #blocked-on-schema-change <https://phabricator.wikimedia.org/tag/blocked-on-schema-change/> t

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-27 Thread Marostegui
Marostegui added a comment. In T246415#6493651 <https://phabricator.wikimedia.org/T246415#6493651>, @Ladsgroup wrote: > - Finding out how much of queries are coming from the API appservers (like ratio of open connections) can be done by DBAs I assume, I can't find any other wa

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-24 Thread Marostegui
Marostegui added a comment. So, I have captured a lots of queries involving `wb_changes` and I haven't found any single query that has a crazy query plan as a result of deleting `wb_changes_change_type wb_changes_change_object_id wb_changes_change_user_id`. Also, I have checked if those

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-23 Thread Marostegui
Marostegui added a comment. @Ladsgroup after a few hours, I have not seen any significant impact on the host's slow queries for now. The host dashboard at https://grafana.wikimedia.org/d/00273/mysql?orgId=1=now-24h=now=db2084=9104 also doesn't show any spikes on anything (scanned rows

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-23 Thread Marostegui
Marostegui added a comment. db2084 is fully repooled, let's monitor its performance TASK DETAIL https://phabricator.wikimedia.org/T262856 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Kormat, Marostegui, Lucas_Werkmeister_WMDE

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-23 Thread Marostegui
Marostegui added a comment. @Ladsgroup db2084 has half the weight it used to, I am capturing live queries arriving to `wb_changes`, so far I haven't found anything strange with their query plans, or extremely slow queries showing up. TASK DETAIL https://phabricator.wikimedia.org/T262856

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-23 Thread Marostegui
Marostegui added a comment. root@db2084.codfw.wmnet[wikidatawiki]> CREATE INDEX /*i*/wb_changes_change_revision_id ON /*_*/wb_changes (change_revision_id); Query OK, 0 rows affected (10.702 sec) Records: 0 Duplicates: 0 Warnings: 0 root@db2084.codfw.wm

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-23 Thread Marostegui
Marostegui added a comment. I can also try to live capture some on db2084. In T262856#6486318 <https://phabricator.wikimedia.org/T262856#6486318>, @Ladsgroup wrote: > Mostly are straightforward but I actually found a query on master in the code that's like th

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-22 Thread Marostegui
Marostegui added a comment. db2084 got those keys dropped: root@db2084.codfw.wmnet[wikidatawiki]> show create table wb_chan

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-21 Thread Marostegui
Marostegui added a project: DBA. Marostegui triaged this task as "Medium" priority. Marostegui claimed this task. Marostegui added a comment. Yep, I will try to pick db2082 which is the one with the less weight. We can also try to capture some of the heaviest queries and replay the

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-21 Thread Marostegui
Marostegui added a comment. In T246415#6477957 <https://phabricator.wikimedia.org/T246415#6477957>, @Ladsgroup wrote: > In T246415#643 <https://phabricator.wikimedia.org/T246415#643>, @Marostegui wrote: > >> Thank you for all this work Amir, this is

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-20 Thread Marostegui
Marostegui added a comment. Thank you for all this work Amir, this is really interesting. I am curious about a few things, and if you'd have the time, can you explain further? > Splitting based on rc table doesn't make sense, let's just drop that. Why doesn't make sense? &

[Wikidata-bugs] [Maniphest] T226093: Capacity planning for Commons Structured Data

2020-09-16 Thread Marostegui
Marostegui added a comment. Following my IRC chat with @ArielGlenn - `revision` and `slots` table on s4 (commonswiki) are still under reasonable sizes. We just decrease the size of the `revision` table as we applied the MCR schema changes there. As of now, the sizes on disk

[Wikidata-bugs] [Maniphest] T262856: Investigate indexes of wb_changes

2020-09-14 Thread Marostegui
Marostegui added a comment. So a quick look (this needs to be taken with care) reveals that some of them are indeed unused: root@db2091.codfw.wmnet[sys]> select * from schema_unused_indexes where object_name='wb_chan

[Wikidata-bugs] [Maniphest] T262187: Wikidata entity dumps didn't start this week

2020-09-07 Thread Marostegui
Marostegui added a comment. Sorry for breaking it! I thought s8 would behave as any other sX section and if you leave a section with no members, MW would automatically pick any other host. Normally, when eqiad is the active DC, we try no to leave any section without any hosts, so we can

[Wikidata-bugs] [Maniphest] T237102: Increase the size of wbt_text_in_lang.wbxl_language

2020-09-03 Thread Marostegui
Marostegui changed the task status from "Stalled" to "Open". Marostegui added a comment. The master was altered finally, so this is unblocked TASK DETAIL https://phabricator.wikimedia.org/T237102 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/pa

[Wikidata-bugs] [Maniphest] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2020-09-03 Thread Marostegui
Marostegui updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Akuckartz, Iflorez

[Wikidata-bugs] [Maniphest] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2020-09-03 Thread Marostegui
Marostegui closed this task as "Resolved". Marostegui added a comment. The master is finally done root@db1109.eqiad.wmnet[wikidatawiki]> ALTER TABLE /*_*/wbt_text_in_lang MODIFY wbxl_language VARBINARY(20) NOT NULL; Query OK, 524052382 rows affected (1 hour 56 m

[Wikidata-bugs] [Maniphest] T237102: Increase the size of wbt_text_in_lang.wbxl_language

2020-09-03 Thread Marostegui
Marostegui closed subtask T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T237102 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup

[Wikidata-bugs] [Maniphest] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2020-09-02 Thread Marostegui
Marostegui changed the task status from "Stalled" to "Open". Marostegui moved this task from Blocked external/Not db team to In progress on the DBA board. TASK DETAIL https://phabricator.wikimedia.org/T237120 WORKBOARD https://phabricator.wikimedia.org/project/board/1060

[Wikidata-bugs] [Maniphest] T237102: Increase the size of wbt_text_in_lang.wbxl_language

2020-09-02 Thread Marostegui
Marostegui changed the status of subtask T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language from Stalled to Open. TASK DETAIL https://phabricator.wikimedia.org/T237102 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2020-08-28 Thread Marostegui
Marostegui changed the status of subtask T239238: Switchover s8 primary database master db1109 - db1104 - Date TBD from Stalled to Open. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing

2020-08-25 Thread Marostegui
Marostegui claimed this task. Marostegui added a subscriber: aaron. Marostegui moved this task from Triage to Blocked external/Not db team on the DBA board. TASK DETAIL https://phabricator.wikimedia.org/T254688 WORKBOARD https://phabricator.wikimedia.org/project/board/1060/ EMAIL

[Wikidata-bugs] [Maniphest] T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing

2020-08-25 Thread Marostegui
Marostegui added a parent task: T250666: Upgrade WMF database-and-backup-related hosts to buster. TASK DETAIL https://phabricator.wikimedia.org/T254688 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron, Marostegui Cc: daniel, Lydia_Pintscher

[Wikidata-bugs] [Maniphest] T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing

2020-08-25 Thread Marostegui
Marostegui added a project: DBA. Marostegui changed the task status from "Open" to "Stalled". Marostegui added a comment. From what I can see the optimizer is chosen either `PRIMARY` or `page_len` depending on the version and the query speeds entirely depends on the Mari

[Wikidata-bugs] [Maniphest] T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing

2020-08-24 Thread Marostegui
Marostegui added a project: mariadb-optimizer-bug. Marostegui added a comment. Unfortunately hard to tell, but the optimizer behaves differently when there is lots of data and sometimes it does silly things :-(. We have seen over the years, especially with massive tables. The only

[Wikidata-bugs] [Maniphest] T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing

2020-08-24 Thread Marostegui
Marostegui added a comment. In T254688#6394904 <https://phabricator.wikimedia.org/T254688#6394904>, @daniel wrote: > Back to the inbox for triage. This isn't actionable as it is. We'd probably have to design a new schema for representing the relevant info in the DB to make

[Wikidata-bugs] [Maniphest] [Created] T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing

2020-06-07 Thread Marostegui
Marostegui created this task. Marostegui added a project: Wikidata. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION The following query takes hours to run and will never run as they get killed by the query killer: Copying to tmp table on diskSELECT

[Wikidata-bugs] [Maniphest] [Updated] T245989: Wikidata API fails with timeout when asking for 5 RC redirects

2020-05-29 Thread Marostegui
Marostegui removed a project: DBA. TASK DETAIL https://phabricator.wikimedia.org/T245989 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, Anomie, Aklapper, pywikibot-bugs-list, Dvorapa, JohnsonLee01, Naike, Dijkstra

[Wikidata-bugs] [Maniphest] [Commented On] T245989: Wikidata API fails with timeout when asking for 5 RC redirects

2020-05-29 Thread Marostegui
Marostegui added a comment. On 10.4 the optimizer keeps choosing the wrong plan, so we need to fix this in code with an `ignore index (page_redirect_namespace_len)` as discussed at T245989#5912205 <https://phabricator.wikimedia.org/T245989#5912205> Removing the #dba

[Wikidata-bugs] [Maniphest] [Commented On] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run

2020-05-27 Thread Marostegui
Marostegui added a comment. The above patch to remove the cronjob has been merged, and the cronjob manually removed from mwmaint1002. I guess this can be resolved @Ladsgroup? TASK DETAIL https://phabricator.wikimedia.org/T238199 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Commented On] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run

2020-05-27 Thread Marostegui
Marostegui added a comment. And this just happened again: | 1953784190 | wikiadmin| 10.64.16.77:53846 | wikidatawiki | Query | 44238 | Copying to tmp table on disk | SELECT /* SpecialFewestRevisions::reallyDoQuery

[Wikidata-bugs] [Maniphest] [Unblock] T208425: [EPIC] Kill the wb_terms table

2020-05-18 Thread Marostegui
Marostegui closed subtask T248086: Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki) as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T208425 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Updated] T252952: Wikidata dispatching slow and maxlag high on Wikidata due to db1101 replication lag

2020-05-16 Thread Marostegui
Marostegui added a comment. Some more details. There were a few long running queries | 669154401 | wikiuser| 10.64.0.59:40112 | wikidatawiki | Query | 95780 | Sending data | SELECT /* SpecialRecentChanges::doMainQuery

[Wikidata-bugs] [Maniphest] [Commented On] T245818: Audit query pages on Wikidata

2020-05-13 Thread Marostegui
Marostegui added a comment. I have raised T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run <https://phabricator.wikimedia.org/T238199> to high as it has caused again errors in production. We need that cronjob either fixed or disabled - everytime it runs w

[Wikidata-bugs] [Maniphest] [Triaged] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run

2020-05-13 Thread Marostegui
Marostegui triaged this task as "High" priority. Marostegui added a comment. This has happened again and caused errors again. Can we please disable it? TASK DETAIL https://phabricator.wikimedia.org/T238199 EMAIL PREFERENCES https://phabricator.wikimedia.org/sett

[Wikidata-bugs] [Maniphest] [Updated] T246415: Investigate a different db load groups for wikidata / wikibase

2020-05-07 Thread Marostegui
Marostegui removed a project: DBA. Marostegui added a comment. Removing the #DBA <https://phabricator.wikimedia.org/tag/dba/> tag as there is nothing for us at the moment. Staying subscribed to the task though, in case I am needed Feel free to add the tag back when needed though.

[Wikidata-bugs] [Maniphest] [Commented On] T251457: LoadBalancer: Transaction spent [n] second(s) in writes, exceeding the limit of [n]

2020-04-30 Thread Marostegui
Marostegui added a comment. Quick answer from my side: no master switch has been done in the last few months on any of our s1-s8, x1 sections TASK DETAIL https://phabricator.wikimedia.org/T251457 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] [Updated] T251457: LoadBalancer: Transaction spent [n] second(s) in writes, exceeding the limit of [n]

2020-04-30 Thread Marostegui
Marostegui removed a project: DBA. TASK DETAIL https://phabricator.wikimedia.org/T251457 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, Liuxinyu970226, brennen, Bugreporter, LarsWirzenius, Hogue, Aklapper, Naike

[Wikidata-bugs] [Maniphest] [Commented On] T251457: LoadBalancer: Transaction spent [n] second(s) in writes, exceeding the limit of [n]

2020-04-30 Thread Marostegui
Marostegui added a comment. This is preventing writes from happening with that particular action,so it must indeed be fixed or whatever caused that increase on time, reverted I would say. TASK DETAIL https://phabricator.wikimedia.org/T251457 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] [Commented On] T251457: LoadBalancer: Transaction spent [n] second(s) in writes, exceeding the limit of [n]

2020-04-30 Thread Marostegui
Marostegui added a comment. It is just a big transaction that takes more than the limit we have for writes, which is 3 seconds. Nothing we can do from the DB infra. TASK DETAIL https://phabricator.wikimedia.org/T251457 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Commented On] T245818: Audit query pages on Wikidata

2020-04-29 Thread Marostegui
Marostegui added a comment. This one T238199 <https://phabricator.wikimedia.org/T238199> is the one I have had to kill the most lately. I also recall MostLinked being pretty heavy as well TASK DETAIL https://phabricator.wikimedia.org/T245818 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T245818: Audit query pages on Wikidata

2020-04-27 Thread Marostegui
Marostegui added a comment. I had to kill `updateSpecialPages.php wikidatawiki --override --only=Fewestrevisions` as it was causing issues - tracking task: T238199 <https://phabricator.wikimedia.org/T238199> TASK DETAIL https://phabricator.wikimedia.org/T245818 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run

2020-04-27 Thread Marostegui
Marostegui added a comment. This hit us hard again: | 1593977753 | wikiadmin | 10.64.16.77:59698 | wikidatawiki | Query | 22436 | Copying to tmp table on disk | SELECT /* SpecialFewestRevisions::reallyDoQuery

[Wikidata-bugs] [Maniphest] [Updated] T250555: test.wikimedia.beta.wmflabs.org pages not loading: Unknown database 'testwikidatawiki'

2020-04-18 Thread Marostegui
Marostegui removed a project: DBA. TASK DETAIL https://phabricator.wikimedia.org/T250555 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Reedy, Aklapper, AlexisJazz, darthmon_wmde, DannyS712, Nandana, Analytics.mediafiles, Lahi, Gq86

[Wikidata-bugs] [Maniphest] [Commented On] T249565: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views

2020-04-14 Thread Marostegui
Marostegui added a comment. Can this task be closed is there anything else left? Apart from finishing the IR? TASK DETAIL https://phabricator.wikimedia.org/T249565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, Marostegui Cc: Risker

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-04-08 Thread Marostegui
Marostegui added a comment. This is all done. All labsdb hosts show this behaviour: root@cumin1001:/home/marostegui# for i in wikidatawiki_p testwikidatawiki_p commonswiki_p testcommonswiki_p; do echo $i; mysql.py -hlabsdb1011 $i -e "select * from wb_terms_no_longer_updated li

[Wikidata-bugs] [Maniphest] [Closed] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-04-08 Thread Marostegui
Marostegui closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T248592 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, Marostegui Cc: Lea_Lacroix_WMDE, JAllemandou, Bstorm, Marostegui, WMDE-leszek

[Wikidata-bugs] [Maniphest] [Updated] T249565: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views

2020-04-08 Thread Marostegui
Marostegui added a subtask: T249683: Redefine mysql GRANTs for wikiadmin. TASK DETAIL https://phabricator.wikimedia.org/T249565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, Marostegui Cc: jcrespo, CDanis, Marostegui, Darwinius

[Wikidata-bugs] [Maniphest] [Updated] T157651: sql.php runs LoadExtensionSchemaUpdates

2020-04-08 Thread Marostegui
Marostegui added a subtask: T249683: Redefine mysql GRANTs for wikiadmin. TASK DETAIL https://phabricator.wikimedia.org/T157651 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Jdforrester-WMF, kostajh, WMDE-leszek, Addshore, Anomie

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-04-06 Thread Marostegui
Marostegui added a comment. Followed up in the patch. Thank you! TASK DETAIL https://phabricator.wikimedia.org/T248592 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, Marostegui Cc: Lea_Lacroix_WMDE, JAllemandou, Bstorm, Marostegui

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-04-06 Thread Marostegui
Marostegui added a comment. ok - let's wait for @Bstorm so we can execute it! :) TASK DETAIL https://phabricator.wikimedia.org/T248592 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, Marostegui Cc: Lea_Lacroix_WMDE, JAllemandou, Bstorm

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-04-06 Thread Marostegui
Marostegui added a comment. So I guess we have to coordinate the following: - @Marostegui renames the `wb_terms` table and create a `wb_terms` empty table - @Bstorm changes `wb_terms_no_longer_updated ` view to point to the renamed version of `wb_terms` which still contains data

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-04-06 Thread Marostegui
Marostegui added a comment. In T248592#6031028 <https://phabricator.wikimedia.org/T248592#6031028>, @Addshore wrote: > As the labs relplica db tables do not get exposed to users at any level I don't know if we should bother renaming them. > If people agree with that state

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-04-05 Thread Marostegui
Marostegui added a comment. Let me know how you guys want to proceed with this and when :) TASK DETAIL https://phabricator.wikimedia.org/T248592 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, Marostegui Cc: JAllemandou, Bstorm

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-03-26 Thread Marostegui
Marostegui added a comment. @Addshore maybe this is not the right task to ask for it but...is Analytics aware of this change? They use labsdb1012 to sqoop data the first days of the month. TASK DETAIL https://phabricator.wikimedia.org/T248592 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-03-26 Thread Marostegui
Marostegui added a comment. That sounds good to me Thank you :) TASK DETAIL https://phabricator.wikimedia.org/T248592 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Bstorm, Marostegui Cc: Marostegui, WMDE-leszek, Aklapper, Addshore, Alter

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-03-26 Thread Marostegui
Marostegui added a comment. So as far as I understand and from https://gerrit.wikimedia.org/r/583693 The view `wb_terms_no_longer_updated` will point to `wb_terms` The current `wb_terms` view points to `wb_terms`. I thought you wanted to, at some point, truncate `wb_terms

[Wikidata-bugs] [Maniphest] [Commented On] T248592: Move wb_terms data in cloud replicas to wb_terms_no_longer_updated

2020-03-26 Thread Marostegui
Marostegui added a comment. @Addshore the last step on Monday I guess you meant create an empty `table`? Also when do you want the table rename to happen? TASK DETAIL https://phabricator.wikimedia.org/T248592 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Phame] [Commented On] Coming to Terms with Change

2020-03-23 Thread Marostegui
Marostegui added a comment. Thanks for this great work!!! POST DETAIL https://phabricator.wikimedia.org/phame/post/view/195/coming_to_terms_with_change/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, Marostegui, Lea_Lacroix_WMDE

[Wikidata-bugs] [Maniphest] [Updated] T208425: [EPIC] Kill the wb_terms table

2020-03-19 Thread Marostegui
Marostegui removed a project: DBA. TASK DETAIL https://phabricator.wikimedia.org/T208425 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Bugreporter, Ladsgroup, JAllemandou, ArielGlenn, jcrespo, Joe, Gehel, alaa_wmde, Marostegui

[Wikidata-bugs] [Maniphest] [Commented On] T208425: [EPIC] Kill the wb_terms table

2020-03-18 Thread Marostegui
Marostegui added a comment. Following up the conversation on IRC with @Ladsgroup I have renamed `wb_terms` to `T208425_wb_terms` on the following hosts and wikis (all in codfw): root@cumin1001:/home/marostegui# for i in db2106 db2109 db2081; do echo $i; mysql.py -h$i

[Wikidata-bugs] [Maniphest] [Updated] T208425: [EPIC] Kill the wb_terms table

2020-03-18 Thread Marostegui
Marostegui added a project: DBA. TASK DETAIL https://phabricator.wikimedia.org/T208425 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Bugreporter, Ladsgroup, JAllemandou, ArielGlenn, jcrespo, Joe, Gehel, alaa_wmde, Marostegui

[Wikidata-bugs] [Maniphest] [Declined] T207006: Set wb_changes_dispatch ROW_FORMAT=COMPRESSED on install and update

2020-03-18 Thread Marostegui
Marostegui closed this task as "Declined". Marostegui added a comment. Going to decline this for now. We are compressing everything as part of T232446 <https://phabricator.wikimedia.org/T232446> TASK DETAIL https://phabricator.wikimedia.org/T207006 EMAIL PRE

[Wikidata-bugs] [Maniphest] [Unblock] T205865: Investigate decrease in wikidata dispatch times due to eqiad -> codfw DC switch

2020-03-18 Thread Marostegui
Marostegui closed subtask T207006: Set wb_changes_dispatch ROW_FORMAT=COMPRESSED on install and update as Declined. TASK DETAIL https://phabricator.wikimedia.org/T205865 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, Marostegui Cc

[Wikidata-bugs] [Maniphest] [Commented On] T208425: [EPIC] Kill the wb_terms table

2020-03-17 Thread Marostegui
Marostegui added a comment. In T208425#5978156 <https://phabricator.wikimedia.org/T208425#5978156>, @Stashbot wrote: > Mentioned in SAL (#wikimedia-releng) [2020-03-18T03:16:34Z] dropping wb_terms table from wikidatawiki in beta cluster (T219123 <https://phabricator.w

[Wikidata-bugs] [Maniphest] [Closed] T239072: SpecialMostRevisions::reallyDoQuery takes lots of hours to run on wikidata

2020-03-11 Thread Marostegui
Marostegui closed this task as "Resolved". Marostegui added a comment. As spoken on IRC: I have removed it manually from mwmaint1002 and run puppet to check it doesn't get added back. TASK DETAIL https://phabricator.wikimedia.org/T239072 EMAIL PREFERENC

[Wikidata-bugs] [Maniphest] [Reopened] T239072: SpecialMostRevisions::reallyDoQuery takes lots of hours to run on wikidata

2020-03-11 Thread Marostegui
Marostegui reopened this task as "Open". Marostegui added a comment. Looks like the cron is still there: # Puppet Name: cron-updatequerypages-mostrevisions-s8@18 0 1 11,25 * * /usr/local/bin/mwscriptwikiset updateSpecialPages.php s8.dblist --override --only=Mostrevisio

[Wikidata-bugs] [Maniphest] [Commented On] T239072: SpecialMostRevisions::reallyDoQuery takes lots of hours to run on wikidata

2020-03-11 Thread Marostegui
Marostegui added a comment. I had to kill this query, as it was hitting db1087 hard again and make it lag F31676132: mysql (6).png <https://phabricator.wikimedia.org/F31676132> TASK DETAIL https://phabricator.wikimedia.org/T239072 EMAIL PREFERENCES https://phabricator.wikimed

[Wikidata-bugs] [Maniphest] [Commented On] T246159: Setting on read new for items causes increased worst response time and number of open connections

2020-02-25 Thread Marostegui
Marostegui added a comment. In T246159#5917713 <https://phabricator.wikimedia.org/T246159#5917713>, @gerritbot wrote: > Change 574863 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): > [mediawiki/extensions/Wikibase@master] Do prefetching entity ids on b

[Wikidata-bugs] [Maniphest] [Commented On] T246005: Wikibase client description API module results in 15k selected rows with new term storage

2020-02-25 Thread Marostegui
Marostegui added a comment. In T246005#5916080 <https://phabricator.wikimedia.org/T246005#5916080>, @Ladsgroup wrote: > https://grafana.wikimedia.org/d/00273/mysql?orgId=1=eqiad%20prometheus%2Fops=db1101=13318=23=1582634607030=1582645401114 > F31630514: imag

[Wikidata-bugs] [Maniphest] [Updated] T245989: Wikidata API fails with timeout when asking for 5 RC redirects

2020-02-24 Thread Marostegui
Marostegui added a project: mariadb-optimizer-bug. Marostegui added a comment. From what I have seen, the optimizer chooses the wrong index on all s8 hosts. I have also tried to optimize `page` to see if refreshing stats would make any different, but it doesn't. The optimizer is silly

[Wikidata-bugs] [Maniphest] [Commented On] T243871: Long query running on dbstore1005:3318

2020-01-31 Thread Marostegui
Marostegui added a comment. After the query was killed no more replication lag has happened F31542138: dbstore.png <https://phabricator.wikimedia.org/F31542138> TASK DETAIL https://phabricator.wikimedia.org/T243871 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings

[Wikidata-bugs] [Maniphest] [Commented On] T243871: Long query running on dbstore1005:3318

2020-01-29 Thread Marostegui
Marostegui added a comment. Thanks - do you want me to kill the query? It is still running there. TASK DETAIL https://phabricator.wikimedia.org/T243871 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Addshore, Ladsgroup, elukey

[Wikidata-bugs] [Maniphest] [Updated] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2020-01-21 Thread Marostegui
Marostegui added a parent task: T243318: 2020 Q3 (or later) codfw -> eqiad switchback. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, jcrespo, alaa_wmde, Addsh

[Wikidata-bugs] [Maniphest] [Commented On] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2020-01-13 Thread Marostegui
Marostegui added a comment. We might be doing a DC switchover in Q3, so we can wait till eqiad is passive to address the final alter on the current primary master. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Commented On] T229407: Spikes in DB traffic and rows/s reads when reading from new terms store

2019-12-09 Thread Marostegui
Marostegui added a comment. Amazing investigation @Ladsgroup! If I had to choose from the two options you provided to solve this, I would go for the second: "Making it not depend on the ParserOutput HTML", as it looks like a better long term solution. Thanks a lot for this w

[Wikidata-bugs] [Maniphest] [Updated] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-26 Thread Marostegui
Marostegui added a subtask: T239238: Switchover s8 primary database master db1109 -> db1104 - Date TBD. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, jcre

[Wikidata-bugs] [Maniphest] [Updated] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-26 Thread Marostegui
Marostegui added a comment. Probably a good idea to coordinate this required failover with: T238966 <https://phabricator.wikimedia.org/T238966> as that one will also probably require a failover as the revision table is quite big TASK DETAIL https://phabricator.wikimedia.org/T237120

[Wikidata-bugs] [Maniphest] [Blocker] T237102: Increase the size of wbt_text_in_lang.wbxl_language

2019-11-26 Thread Marostegui
Marostegui changed the status of subtask T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language from Open to Stalled. TASK DETAIL https://phabricator.wikimedia.org/T237102 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Changed Status] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-26 Thread Marostegui
Marostegui changed the task status from "Open" to "Stalled". Marostegui updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui

[Wikidata-bugs] [Maniphest] [Commented On] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-26 Thread Marostegui
Marostegui added a comment. Impossible to alter this table on the master while it is a master. I just got a bunch of metadata locking for lots of connections and the cluster had a glitch. We'd need to schedule a master failover for s8 after Christmas in order to get the master done. TASK

[Wikidata-bugs] [Maniphest] [Created] T239072: SpecialMostRevisions::reallyDoQuery takes lots of hours to run on wikidata

2019-11-25 Thread Marostegui
Marostegui created this task. Marostegui added projects: Wikidata, MediaWiki-Special-pages, Core Platform Team. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION The following query runs on the vslow host of `wikidatawiki` (which is good): root@db1087.eqiad.wmnet

[Wikidata-bugs] [Maniphest] [Commented On] T68025: [Story] Monitor size of some Wikidata database tables

2019-11-20 Thread Marostegui
Marostegui added a comment. Thanks @Addshore! I think the size as @Ladsgroup points out could be pretty cool. Not sure if we should add totals and/or just data+indexes separately, what do you guys think? I am definitely interested on the total size for sure (data+indexes), but not sure

[Wikidata-bugs] [Maniphest] [Edited] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-19 Thread Marostegui
Marostegui updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Iflorez, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Edited] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-18 Thread Marostegui
Marostegui updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Iflorez, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Commented On] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-18 Thread Marostegui
Marostegui added a comment. s8 eqiad progress [ ] labsdb1012 [ ] labsdb1011 [ ] labsdb1010 [ ] labsdb1009 [ ] dbstore1005 [ ] db1126 [ ] db1124 [ ] db1116 [ ] db1109 [ ] db1104 [ ] db1101 [ ] db1099 [ ] db1092 [ ] db1087 TASK DETAIL https

[Wikidata-bugs] [Maniphest] [Commented On] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-18 Thread Marostegui
Marostegui added a comment. This is how the table looks like after the ALTER: Table: wbt_text_in_lang Create Table: CREATE TABLE `wbt_text_in_lang` ( `wbxl_id` int(10) unsigned NOT NULL AUTO_INCREMENT, `wbxl_language` varbinary(20) NOT NULL, `wbxl_text_id

[Wikidata-bugs] [Maniphest] [Edited] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-18 Thread Marostegui
Marostegui updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Iflorez, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Edited] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-18 Thread Marostegui
Marostegui updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Iflorez, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Unblock] T194163: Drop change_tag.ct_tag column

2019-11-14 Thread Marostegui
Marostegui closed subtask T210713: Drop change_tag.ct_tag column in production as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T194163 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, Marostegui Cc: gerritbot, Aklapper

[Wikidata-bugs] [Maniphest] [Created] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run

2019-11-13 Thread Marostegui
Marostegui created this task. Marostegui added projects: Wikidata, MediaWiki-General. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION I just realised that we had a wikidatawiki vslow host with the following query: {P9611} It was running for more than 9h, it was creating

[Wikidata-bugs] [Maniphest] [Updated] T226093: Capacity planning for Commons Structured Data

2019-11-12 Thread Marostegui
Marostegui added a comment. For reference: T232446#5657140 <https://phabricator.wikimedia.org/T232446#5657140> TASK DETAIL https://phabricator.wikimedia.org/T226093 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: Maro

[Wikidata-bugs] [Maniphest] [Commented On] T226093: Capacity planning for Commons Structured Data

2019-11-12 Thread Marostegui
Marostegui added a comment. I would like to know if there is some work going on to be able to split those tables from s4 into their own set of servers. My understanding is that it wasn't possible and that's why the tables were created on s4 (where Commons live) directly. Sharing the same

[Wikidata-bugs] [Maniphest] [Commented On] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language

2019-11-11 Thread Marostegui
Marostegui added a comment. I can try to start with this next week to make sure the deletion of wb_terms doesn't get (more) blocked. I have a huge backlog to catch up with after my holidays so impossible to start with this during this week. testwikidata should be easy to do and I will do

[Wikidata-bugs] [Maniphest] [Commented On] T230862: Create a way to filter only WB-related changes from Commons recentchanges

2019-10-17 Thread Marostegui
Marostegui added a comment. In T230862#5538510 <https://phabricator.wikimedia.org/T230862#5538510>, @Anomie wrote: > Using tags has the advantage of more directly identifying the relevant revisions, //if// the planner decides that gathering all revisions with the tag then

  1   2   3   4   5   6   7   >