[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup closed this task as "Resolved". Ladsgroup added a project: DBA. Ladsgroup claimed this task. Ladsgroup added a comment. One month have passed. We haven't had major issues like we used to (dumps running making it impossible to depool any host in s8) so I call this done. We can

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. In T138208#7844298 , @Ladsgroup wrote: > It's a bit hard to measure but it's probably fixed. That would be wonderful if true. Let's leave this open for a while yet just in case... TASK DETAIL

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. It's a bit hard to measure but it's probably fixed. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

gerritbot added a comment. Change 767477 **merged** by jenkins-bot: [operations/dumps@master] Add --dbgroupdefault=dump to every major dump run https://gerrit.wikimedia.org/r/767477 TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. I'm surprised --dbgroupdefault is not set in any dump run, it's a built-in system in mediawiki to provide support for cases like this. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

gerritbot added a comment. Change 767477 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani): [operations/dumps@master] Add --dbgroupdefault=dump to every major dump run https://gerrit.wikimedia.org/r/767477 TASK DETAIL

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. In T138208#7568960 , @ArielGlenn wrote: > As I feared, fetchText.php calls MediaWikiServices::getInstance()->getBlobStore()->getBlob() which gets a db replica connection on its own, with no opportunity for us

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

daniel added a comment. In T138208#7727286 , @Ladsgroup wrote: > At least it can avoid connecting to non-dump hosts. I thought we did that... maybe @ArielGlenn has an idea. TASK DETAIL https://phabricator.wikimedia.org/T138208

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. I am aware of and following this discussion but right now, my responsiveness on this task will be slow, most of my time needs to go to getting my teammate who will be dumps co-maintainer up to speed. Please bear with us. TASK DETAIL

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. At least it can avoid connecting to non-dump hosts. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup Cc: LSobanski, Ladsgroup, Marostegui, Addshore,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

daniel added a comment. In T138208#7727264 , @Ladsgroup wrote: > Possibly but also keeping the connection open? Maybe it needs to buffer, close the connection and then compress given that it's cpu intensive and slow? WikiExporter

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. Possibly but also keeping the connection open? Maybe it needs to buffer, close the connection and then compress given that it's cpu intensive and slow? TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

daniel added a comment. In T138208#7727235 , @Ladsgroup wrote: > Most notably the top of the snapshot is not the dumper, it's bzip2. Does it keep connection open while compressing? Isn't it compressing the stream on the fly? TASK

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. I'm fairly certain it does. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. Most notably the top of the snapshot is not the dumper, it's bzip2. Does it keep connection open while compressing? TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. Today, I was trying to upgrade s8 to bullseye and I can't depool any host, they all ended up with lingering connections from snapshot1011.eqiad.wmnet. For the whole hour in the hosts there, it was just sleep. Looking at snapshot1011, the dumper is

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. Thanks. I was pretty careful with my testing for the last fix, making sure that in production the patch redirected to a vslow/dump server. But I may have overlooked something. :-( TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a comment. Next time I will try a different approach to capture them, which I think it will give us the exact queries TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a comment. I tried but I wasn't able to capture any :( TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: LSobanski, Ladsgroup, Marostegui, Addshore,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. I hate to ask but can we capture any queries? TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArielGlenn Cc: LSobanski, Ladsgroup, Marostegui, Addshore,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a comment. So I am still seeing a wikiadmin connection from snapshot1011.eqiad.wmnet. to a s8 API host (db1172) TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc:

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. The above patch was deployed with the train everywhere, so the specific set of queries should no longer be directed to non-vslow/dump db servers. If that's the cas, we are now back to the harder issue of what to do when a db server is depooled, and I think that

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

daniel added a comment. In T138208#7625170 , @Ladsgroup wrote: > I think we have different perspectives and it might be because I'm coming from SRE? I personally think dumps are actually the environment, not the code. I see that

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.18; 2022-01-17). TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ReleaseTaggerBot Cc: LSobanski, Ladsgroup, Marostegui, Addshore,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

gerritbot added a comment. Change 747455 **merged** by jenkins-bot: [mediawiki/core@master] try to use 'dump' group for db connections for dumps of page content https://gerrit.wikimedia.org/r/747455 TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. The patch at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/747455/ is tested and ready to go, and in line with the way existing dumps scripts work. So I'd like to go ahead with it. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. I think we have different perspectives and it might be because I'm coming from SRE? I personally think dumps are actually the environment, not the code. Beside the maint. script itself, they basically share the same code as Special:Export and that is being used

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

daniel added a comment. In T138208#7612881 , @Ladsgroup wrote: > Thanks, that was the missing piece. My suggestion would be to set an environmental variable in calling mw scripts (if it's not set already). phpunit does a similar thing.

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. Thanks, that was the missing piece. My suggestion would be to set an environmental variable in calling mw scripts (if it's not set already). phpunit does a similar thing. And in LB code in mw when trying to get a connection, it should check the env variable and

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. There is a complicated set of python scripts that coordinate the dump jobs for each wiki during the two monthly runs. https://wikitech.wikimedia.org/wiki/Dumps/Current_Architecture gives an overview.

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. More on operations part of it, how they are being ran? Is it cron/systemd timer? if so where? What is the maint script or bash script that is being ran, etc. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. In T138208#7611718 , @Ladsgroup wrote: > In T138208#7611712 , @ArielGlenn wrote: > >> Not yet; I need to talk with someone more knowledgeable than me

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Ladsgroup added a comment. In T138208#7611712 , @ArielGlenn wrote: > Not yet; I need to talk with someone more knowledgeable than me about whether this approach is reasonable, before moving forward. I'll bring it up at our next meeting

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. In T138208#7611708 , @Marostegui wrote: > In T138208#7571559 , @gerritbot wrote: > >> Change 747455 had a related patch set uploaded (by ArielGlenn;

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a comment. In T138208#7571559 , @gerritbot wrote: > Change 747455 had a related patch set uploaded (by ArielGlenn; author: ArielGlenn): > > [mediawiki/core@master] try to use 'dump' group for db connections for

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

gerritbot added a comment. Change 747455 had a related patch set uploaded (by ArielGlenn; author: ArielGlenn): [mediawiki/core@master] try to use 'dump' group for db connections for dumps of page content https://gerrit.wikimedia.org/r/747455 TASK DETAIL

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. Thanks for this thought, Daniel. I think it's better if I can pass the dbgroupdefault parameter to the maintenance script itself, instead of hacking something into getBlob(). But I do need to check if that's going to work ok. The longer term fix you mentioned, is

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

daniel added a comment. In T138208#7568960 , @ArielGlenn wrote: > As I feared, fetchText.php calls MediaWikiServices::getInstance()->getBlobStore()->getBlob() which gets a db replica connection on its own, with no opportunity for us to

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a comment. Thanks for looking into this issue :-) TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. As I feared, fetchText.php calls MediaWikiServices::getInstance()->getBlobStore()->getBlob() which gets a db replica connection on its own, with no opportunity for us to ask that it be in the vslow/dump group. We might be able to use the -dbgroupdefault dump

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. The above is happening from pages-meta-history dumps, and I will look into it later today. The snapshot1008 (wikidata entity) dumps will be harder. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a comment. Following my chat with @ArielGlenn on IRC, I got this query coming from snapshot1011 2690453130wikiadmin10.64.0.156:38912wikidatawikiQuery0 StatisticsSELECT /* MediaWiki\\Storage\\SqlBlobStore::fetchBlobs dumpsgen@snapsh... */

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. The reason only those two snapshot hosts are involved is undoubtedly because dumps on the others have finished for this run. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a comment. Now that I look into it, the `wikiadmin` user is connected to all the pooled hosts, regardless of their config group. Connections are from snapshost1008 and snapshost1011 This would make impossible to run an automated schema change on any of those wikidata

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a subscriber: Ladsgroup. Marostegui added a comment. This keeps handling and it is painful if we want to run automated schema changes (T288235 ). I have tried to deploy a change on s8 without manual intervention but it got stuck on:

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Addshore closed subtask T138381: Allow DB group used by ChangeDispatcher to be configured as Declined. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Marostegui, Addshore,

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. mysql.php, used for wikidata entity dumps, does not apparently correctly handle the --group flag. it's unclear to me what it does do, I need to check into this sometime later. The queries run by it are extremely short so the impact is minimal, but it still needs

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. In T138208#6811418 , @Addshore wrote: > In T138208#6809784 , @ArielGlenn wrote: > >> This is because the maintenance scripts that do "small" page ranges

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Addshore moved this task from Discussion to External Realm on the wdwb-tech-focus board. Addshore added a comment. In T138208#6809784 , @ArielGlenn wrote: > This is because the maintenance scripts that do "small" page ranges take

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

ArielGlenn added a comment. This is because the maintenance scripts that do "small" page ranges take several hours to complete. I will keep this in mind for when we can go to multiple bz2 streams in the page content history dumps; I'll be able to dump much smaller ranges then and concat

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Marostegui added a comment. I depooled today db (https://phabricator.wikimedia.org/P14229) which is not on the vslow,dump group and I still see two things: - The host is being accessed by `wikiadmin` from `snapshot1006`: - The host keeps being accessed even though it's been

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

jcrespo added a subscriber: Marostegui. jcrespo added a comment. Handover to @Marostegui for him to comment, as he will be the person to know if this continues happening or now. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

Addshore moved this task from Backlog to Discussion on the wdwb-tech-focus board. Addshore added a comment. Trying to decode if there are any further concrete actionables here and who needs to action them? TASK DETAIL https://phabricator.wikimedia.org/T138208 WORKBOARD