[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Lucas_Werkmeister_WMDE closed this task as "Resolved". Lucas_Werkmeister_WMDE moved this task from Tech Verification to Our work done on the Wikidata Dev Team (Sprint-♾) board. Lucas_Werkmeister_WMDE added a comment. I don’t think we can verify this, let’s just assume it’s working better now

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Manuel edited projects, added Wikidata Dev Team (Sprint-♾); removed Wikidata-Campsite (Team A Hearth ). TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo, Manuel Cc: Mohammed_Sadat_WMDE,

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo, Maintenance_bot Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael,

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

gerritbot added a comment. Change 768032 **merged** by Ladsgroup: [operations/puppet@production] Wikibase dumps: Lower batch size (reduce run time) https://gerrit.wikimedia.org/r/768032 TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo, gerritbot Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael, Lucas_Werkmeister_WMDE, hoo,

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

gerritbot added a comment. Change 768032 had a related patch set uploaded (by Hoo man; author: Hoo man): [operations/puppet@production] Wikibase dumps: Lower batch size (reduce run time) https://gerrit.wikimedia.org/r/768032 TASK DETAIL https://phabricator.wikimedia.org/T300255

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Mohammed_Sadat_WMDE added a comment. made a note about the dump fail in the next weekly summary . TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Lucas_Werkmeister_WMDE triaged this task as "High" priority. Lucas_Werkmeister_WMDE added a comment. If the DBAs are going to kill our dumps over this then I assume finding a solution must be high priority. TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Ladsgroup added a comment. I'm sorry but I just killed all rdf and json dumper that kept connecting to depooled s8 db I needed to do maintenance on. Please assume this week's dump is failed. TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-03-02T08:02:20Z] killing all entity dumpers of wikidata in snapshot1008 (T300255 ) TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Ladsgroup added a comment. NVM, it is the xml dumper T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Ladsgroup added a comment. Today, I was trying to upgrade s8 to bullseye and I can't depool any host, they all ended up with lingering connections from `snapshot1011.eqiad.wmnet`. Can you in the meantime tell me when this script is being ran? so I avoid that time. TASK DETAIL

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

hoo claimed this task. hoo added a project: Wikidata-Campsite (Team A Hearth ). hoo added a comment. I'll have a look soon (probably in early March, as I'm away-ish now). TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Lucas_Werkmeister_WMDE added a comment. Okay, that makes sense. So we’d have the same number of dumpers running in parallel, each of them sequentially running through its segment of the item IDs with periodic restarts. TASK DETAIL https://phabricator.wikimedia.org/T300255 EMAIL

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Ladsgroup added a comment. > Hm, this would be visible in the dump files, right? I don't think so, maybe I'm missing your point. what it should be as something along the lines of other dumpers, short period of time and then picking it up from where it was left off. So after doing

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Lucas_Werkmeister_WMDE added a comment. Hm, this would be visible in the dump files, right? I remember noticing a while ago that the dump started with a subset of item IDs in ascending order, then eventually jumped back to the beginning and did another subset, and so on. If I understand

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

Ladsgroup created this task. Ladsgroup added projects: wdwb-tech, Wikidata. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION I have depooled a replica in es5 to do maintenance and after an hour, there are still connections to it from the entity data dumper. That's the only