[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-11-24 Thread Michael
Michael removed Michael as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael Cc: ArielGlenn, Michael, Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-11-17 Thread Ladsgroup
Ladsgroup added a comment. In T246415#6616074 , @Addshore wrote: > In T246415#6541422 , @Ladsgroup wrote: > >> In T246415#6536970

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-11-10 Thread Addshore
Addshore added a comment. In T246415#6541422 , @Ladsgroup wrote: > In T246415#6536970 , @Addshore wrote: > >> Looking at the ACs I think we have some open questions still >> >>

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-11-04 Thread ArielGlenn
ArielGlenn added a project: User-ArielGlenn. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael, ArielGlenn Cc: ArielGlenn, Michael, Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-10-14 Thread Ladsgroup
Ladsgroup added a comment. These are my opinions only, if anyone wants to double check, that'd be great. In T246415#6536970 , @Addshore wrote: > Looking at the ACs I think we have some open questions still > > - How feasible is

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-10-12 Thread Addshore
Addshore added a comment. Looking at the ACs I think we have some open questions still - How feasible is it to introduce a "client" and "repo" db load group and have it used in the right places in our code? / how could that be tackled. - How feasible is it to introduce a "terms" db

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-30 Thread Michael
Michael moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞) board. Michael added a comment. I feel this now needs more feedback again before we can think about further steps. TASK DETAIL https://phabricator.wikimedia.org/T246415 WORKBOARD

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-30 Thread Michael
Michael added a comment. So, to make a baby step of moving this ticket forward: Based on the discussion so far I could see two possible ways to maybe split things: Option 1: One extra load group for Term requests only - Reasoning

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-30 Thread Michael
Michael claimed this task. Restricted Application added a project: User-Michael. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael Cc: Michael, Marostegui, Ladsgroup, WMDE-leszek, Aklapper,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-30 Thread noarave
noarave closed subtask T262924: Investigate how to implement different DB load groups in Wikibase as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: noarave Cc: Michael, Marostegui,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-27 Thread Marostegui
Marostegui added a comment. In T246415#6493651 , @Ladsgroup wrote: > - Finding out how much of queries are coming from the API appservers (like ratio of open connections) can be done by DBAs I assume, I can't find any other way to find

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-25 Thread Ladsgroup
Ladsgroup added a comment. Hope these answers help. In T246415#6494603 , @Michael wrote: >> lots of them actually coming from mediawiki core and we have no control over destination of queries originating from there > > But that

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-25 Thread Michael
Michael added a comment. I'm looking into this, trying to understand it and get an idea whether I'm able to contribute something and thus should pick it up. But I'm currently still confused and have some questions: > lots of them actually coming from mediawiki core and we have no

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-25 Thread Ladsgroup
Ladsgroup added a comment. Okay, Let me give on overview on what has been worked on so far, this is basically a summary of the previous comments and reshaped: - Queries to term store are basically half of the load. - Given that it's actually 1% of queries that actually reach the db

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-22 Thread Ladsgroup
Ladsgroup added a comment. Filed: T263520: Updating user edit count is pretty expensive in some wikis TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-22 Thread Ladsgroup
Ladsgroup added a comment. On average 60% of term store queries come from client wikis : F32358730: image.png TASK DETAIL

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-21 Thread Marostegui
Marostegui added a comment. In T246415#6477957 , @Ladsgroup wrote: > In T246415#643 , @Marostegui wrote: > >> Thank you for all this work Amir, this is really interesting. I am

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-21 Thread Ladsgroup
Ladsgroup added a comment. In T246415#643 , @Marostegui wrote: > Thank you for all this work Amir, this is really interesting. I am curious about a few things, and if you'd have the time, can you explain further? > >> Splitting

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-20 Thread Marostegui
Marostegui added a comment. Thank you for all this work Amir, this is really interesting. I am curious about a few things, and if you'd have the time, can you explain further? > Splitting based on rc table doesn't make sense, let's just drop that. Why doesn't make sense? > 56%

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-18 Thread Ladsgroup
Ladsgroup added a comment. While I'm here, I did a quick check on matter of pressure of writes on master. And boy I'm not disappointed: - 6% of all of the write pressure is just this: 95904.001 (6.519%) UPDATE `user` SET `user_editcount` = `user_editcount` + ? WHERE

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-18 Thread Ladsgroup
Ladsgroup added a comment. Top queries that put more than 1% of the total load on the replicas: Termbox: 23690016.0 (41.05%) SELECT `wbtl_id` , `wbtl_type_id` , `wbxl_language` , `wbx_text` , `wbit_item_id` FROM `wbt_term_in_lang` JOIN `wbt_text_in_lang` ON ( (

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-18 Thread Ladsgroup
Ladsgroup added a comment. In T246415#6474189 , @Addshore wrote: >> s8 is coming from clients or replicas. > > clients or replicas? I don't quite follow this bit? s/replicas/repos/ I wrote replicas so many times my brain

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-18 Thread Addshore
Addshore added a comment. > s8 is coming from clients or replicas. clients or replicas? I don't quite follow this bit? TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc:

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-18 Thread Ladsgroup
Ladsgroup added a comment. What we have done here: We went through the performance schema of replicas in codfw since the switchover (September 2, 17 days ago) to see what is the nature of queries going to replicas. Total latency of selects for s8 replicas is 57712491 seconds (roughly 100

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread WMDE-leszek
WMDE-leszek added a comment. @Addshore and @Ladsgroup to share pointers to the existing ways of getting the data possibly relevant for deciding on the desired approach TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread WMDE-leszek
WMDE-leszek added a comment. Task break down to happen with @Addshore TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread Lucas_Werkmeister_WMDE
Lucas_Werkmeister_WMDE moved this task from Prioritized Tech to Wikidata-Campsite-Iteration-∞ on the Wikidata-Campsite board. Lucas_Werkmeister_WMDE edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞); removed Wikidata-Campsite. TASK DETAIL

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread WMDE-leszek
WMDE-leszek updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread WMDE-leszek
WMDE-leszek updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread WMDE-leszek
WMDE-leszek updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread gerritbot
gerritbot added a comment. Change 593335 **abandoned** by Addshore: [mediawiki/extensions/Wikibase@master] WIP DNM DRAFT, potential way to track the DB usage for wikibase Reason: https://gerrit.wikimedia.org/r/593335 TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread Addshore
Addshore updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-09-15 Thread Addshore
Addshore updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule, Beast1978, Un1tY,

[Wikidata-bugs] [Maniphest] T246415: Investigate a different db load groups for wikidata / wikibase

2020-07-17 Thread Addshore
Addshore removed Addshore as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T246415 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Marostegui, Ladsgroup, WMDE-leszek, Aklapper, Addshore, Alter-paule,