[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore closed this task as "Resolved". Addshore claimed this task. Addshore added a comment. Markins as resolved as the investigation part is done and some implementation to make an initial split has happened T281453 TASK DETAIL

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore closed subtask T262923: Investigate sensible DB load group splits for Wikidata / Wikibase as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: ArielGlenn, Michael,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore removed a project: Wikidata-Campsite. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: ArielGlenn, Michael, Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Invadibot,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: ArielGlenn, Michael, Marostegui, Ladsgroup, WMDE-leszek, Aklapper,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

WMDE-leszek edited projects, added Wikidata-Campsite; removed Wikidata-Campsite (Wikidata-Campsite-Iteration-∞). TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: ArielGlenn,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Michael removed Michael as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael Cc: ArielGlenn, Michael, Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. In T246415#6616074 , @Addshore wrote: > In T246415#6541422 , @Ladsgroup wrote: > >> In T246415#6536970

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore added a comment. In T246415#6541422 , @Ladsgroup wrote: > In T246415#6536970 , @Addshore wrote: > >> Looking at the ACs I think we have some open questions still >> >>

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

ArielGlenn added a project: User-ArielGlenn. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael, ArielGlenn Cc: ArielGlenn, Michael, Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. These are my opinions only, if anyone wants to double check, that'd be great. In T246415#6536970 , @Addshore wrote: > Looking at the ACs I think we have some open questions still > > - How feasible is

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore added a comment. Looking at the ACs I think we have some open questions still - How feasible is it to introduce a "client" and "repo" db load group and have it used in the right places in our code? / how could that be tackled. - How feasible is it to introduce a "terms" db

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Michael moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞) board. Michael added a comment. I feel this now needs more feedback again before we can think about further steps. TASK DETAIL https://phabricator.wikimedia.org/T246415 WORKBOARD

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Michael added a comment. So, to make a baby step of moving this ticket forward: Based on the discussion so far I could see two possible ways to maybe split things: Option 1: One extra load group for Term requests only - Reasoning

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Michael claimed this task. Restricted Application added a project: User-Michael. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael Cc: Michael, Marostegui, Ladsgroup, WMDE-leszek, Aklapper,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

noarave closed subtask T262924: Investigate how to implement different DB load groups in Wikibase as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: noarave Cc: Michael, Marostegui,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Marostegui added a comment. In T246415#6493651 , @Ladsgroup wrote: > - Finding out how much of queries are coming from the API appservers (like ratio of open connections) can be done by DBAs I assume, I can't find any other way to find

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. Hope these answers help. In T246415#6494603 , @Michael wrote: >> lots of them actually coming from mediawiki core and we have no control over destination of queries originating from there > > But that

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Michael added a comment. I'm looking into this, trying to understand it and get an idea whether I'm able to contribute something and thus should pick it up. But I'm currently still confused and have some questions: > lots of them actually coming from mediawiki core and we have no

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. Okay, Let me give on overview on what has been worked on so far, this is basically a summary of the previous comments and reshaped: - Queries to term store are basically half of the load. - Given that it's actually 1% of queries that actually reach the db

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. Filed: T263520: Updating user edit count is pretty expensive in some wikis TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. On average 60% of term store queries come from client wikis : F32358730: image.png TASK DETAIL

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Marostegui added a comment. In T246415#6477957 , @Ladsgroup wrote: > In T246415#643 , @Marostegui wrote: > >> Thank you for all this work Amir, this is really interesting. I am

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. In T246415#643 , @Marostegui wrote: > Thank you for all this work Amir, this is really interesting. I am curious about a few things, and if you'd have the time, can you explain further? > >> Splitting

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Marostegui added a comment. Thank you for all this work Amir, this is really interesting. I am curious about a few things, and if you'd have the time, can you explain further? > Splitting based on rc table doesn't make sense, let's just drop that. Why doesn't make sense? > 56%

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. While I'm here, I did a quick check on matter of pressure of writes on master. And boy I'm not disappointed: - 6% of all of the write pressure is just this: 95904.001 (6.519%) UPDATE `user` SET `user_editcount` = `user_editcount` + ? WHERE

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. Top queries that put more than 1% of the total load on the replicas: Termbox: 23690016.0 (41.05%) SELECT `wbtl_id` , `wbtl_type_id` , `wbxl_language` , `wbx_text` , `wbit_item_id` FROM `wbt_term_in_lang` JOIN `wbt_text_in_lang` ON ( (

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. In T246415#6474189 , @Addshore wrote: >> s8 is coming from clients or replicas. > > clients or replicas? I don't quite follow this bit? s/replicas/repos/ I wrote replicas so many times my brain

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore added a comment. > s8 is coming from clients or replicas. clients or replicas? I don't quite follow this bit? TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc:

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Ladsgroup added a comment. What we have done here: We went through the performance schema of replicas in codfw since the switchover (September 2, 17 days ago) to see what is the nature of queries going to replicas. Total latency of selects for s8 replicas is 57712491 seconds (roughly 100

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

WMDE-leszek added a comment. @Addshore and @Ladsgroup to share pointers to the existing ways of getting the data possibly relevant for deciding on the desired approach TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

WMDE-leszek added a comment. Task break down to happen with @Addshore TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Lucas_Werkmeister_WMDE moved this task from Prioritized Tech to Wikidata-Campsite-Iteration-∞ on the Wikidata-Campsite board. Lucas_Werkmeister_WMDE edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞); removed Wikidata-Campsite. TASK DETAIL

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

WMDE-leszek updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

WMDE-leszek updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

WMDE-leszek updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

gerritbot added a comment. Change 593335 **abandoned** by Addshore: [mediawiki/extensions/Wikibase@master] WIP DNM DRAFT, potential way to track the DB usage for wikibase Reason: https://gerrit.wikimedia.org/r/593335 TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

Addshore removed Addshore as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule,