[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE closed this task as "Resolved".
Lucas_Werkmeister_WMDE moved this task from Tech Verification to Our work done 
on the Wikidata Dev Team (Sprint-♾) board.
Lucas_Werkmeister_WMDE added a comment.


  I don’t think we can verify this, let’s just assume it’s working better now 
unless we hear otherwise from the DBAs.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

WORKBOARD
  https://phabricator.wikimedia.org/project/board/6017/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Lucas_Werkmeister_WMDE
Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael, Lucas_Werkmeister_WMDE, 
hoo, LSobanski, Marostegui, Aklapper, Ladsgroup, Astuthiodit_1, karapayneWMDE, 
Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Manuel]

Manuel edited projects, added Wikidata Dev Team (Sprint-♾); removed 
Wikidata-Campsite (Team A Hearth ).

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Manuel
Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael, Lucas_Werkmeister_WMDE, 
hoo, LSobanski, Marostegui, Aklapper, Ladsgroup, Astuthiodit_1, karapayneWMDE, 
Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331, Lydia_Pintscher
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Maintenance_bot
Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael, Lucas_Werkmeister_WMDE, 
hoo, LSobanski, Marostegui, Aklapper, Ladsgroup, karapayneWMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331, Fernandobacasegua34, 786, Suran38, Biggs657, 
Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan, Neuronton
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[gerritbot]

gerritbot added a comment.


  Change 768032 **merged** by Ladsgroup:
  
  [operations/puppet@production] Wikibase dumps: Lower batch size (reduce run 
time)
  
  https://gerrit.wikimedia.org/r/768032

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, gerritbot
Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael, Lucas_Werkmeister_WMDE, 
hoo, LSobanski, Marostegui, Aklapper, Ladsgroup, Fernandobacasegua34, 786, 
Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, 
Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, 
Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, gerritbot
Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael, Lucas_Werkmeister_WMDE, 
hoo, LSobanski, Marostegui, Aklapper, Ladsgroup, Fernandobacasegua34, 786, 
Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, 
Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, 
Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[gerritbot]

gerritbot added a comment.


  Change 768032 had a related patch set uploaded (by Hoo man; author: Hoo man):
  
  [operations/puppet@production] Wikibase dumps: Lower batch size (reduce run 
time)
  
  https://gerrit.wikimedia.org/r/768032

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, gerritbot
Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael, Lucas_Werkmeister_WMDE, 
hoo, LSobanski, Marostegui, Aklapper, Ladsgroup, karapayneWMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Mohammed_Sadat_WMDE]

Mohammed_Sadat_WMDE added a comment.


  made a note about the dump fail in the next weekly summary 
.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Mohammed_Sadat_WMDE
Cc: Mohammed_Sadat_WMDE, ItamarWMDE, Addshore, Michael, Lucas_Werkmeister_WMDE, 
hoo, LSobanski, Marostegui, Aklapper, Ladsgroup, karapayneWMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE triaged this task as "High" priority.
Lucas_Werkmeister_WMDE added a comment.


  If the DBAs are going to kill our dumps over this then I assume finding a 
solution must be high priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Lucas_Werkmeister_WMDE
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, karapayneWMDE, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Ladsgroup]

Ladsgroup added a comment.


  I'm sorry but I just killed all rdf and json dumper that kept connecting to 
depooled s8 db I needed to do maintenance on. Please assume this week's dump is 
failed.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Ladsgroup
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, karapayneWMDE, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2022-03-02T08:02:20Z]  
killing all entity dumpers of wikidata in snapshot1008 (T300255 
)

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Stashbot
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, karapayneWMDE, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Ladsgroup]

Ladsgroup added a comment.


  NVM, it is the xml dumper T138208: Connections to all db servers for wikidata 
as wikiadmin from snapshot, terbium 

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Ladsgroup
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, karapayneWMDE, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Ladsgroup]

Ladsgroup added a comment.


  Today, I was trying to upgrade s8 to bullseye and I can't depool any host, 
they all ended up with lingering connections from `snapshot1011.eqiad.wmnet`. 
Can you in the meantime tell me when this script is being ran? so I avoid that 
time.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo, Ladsgroup
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, karapayneWMDE, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[hoo]

hoo claimed this task.
hoo added a project: Wikidata-Campsite (Team A Hearth ).
hoo added a comment.


  I'll have a look soon (probably in early March, as I'm away-ish now).

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, karapayneWMDE, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.


  Okay, that makes sense. So we’d have the same number of dumpers running in 
parallel, each of them sequentially running through its segment of the item IDs 
with periodic restarts.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, karapayneWMDE, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Ladsgroup]

Ladsgroup added a comment.


  > Hm, this would be visible in the dump files, right?
  
  I don't think so, maybe I'm missing your point.
  
  what it should be as something along the lines of other dumpers, short period 
of time and then picking it up from where it was left off. So after doing maybe 
1000 items, it can exit and signal the last item id it was done, then the 
runner picks it up and start from that item id and run the script again 
concatenating to the same file. It doesn't need to change anything in file 
structure or concurrency of the jobs.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.


  Hm, this would be visible in the dump files, right? I remember noticing a 
while ago that the dump started with a subset of item IDs in ascending order, 
then eventually jumped back to the beginning and did another subset, and so on. 
If I understand correctly, this is because we run eight(?) dumpers in parallel, 
going through item IDs congruent 0-7 modulo 8 (or something like that – it 
might also be based on the page ID), and then we concatenate their results at 
the end (with some slight editing to fix up the JSON array syntax). If we 
increase the number of dumpers, the results will probably be listed in a 
different order.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T300255: Wikidata entity dumper keeps connecting to depooled host for really long time

[Ladsgroup]

Ladsgroup created this task.
Ladsgroup added projects: wdwb-tech, Wikidata.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  I have depooled a replica in es5 to do maintenance and after an hour, there 
are still connections to it from the entity data dumper. That's the only maint 
script/dumper that still holds and keeps connecting to the depooled db after 
this time. This problem hinders DBAs ability to do maintenance (and more 
importantly primary switchovers are not getting reflected meaning the primary 
can get a lot of reads).
  
  This is similar to T298485: MW scripts should reload the database config 
 and a proper fix would be to reload 
config on the fly but at least (and as a quick measure), the dumper should not 
take this long, it should be split to smaller batches that take maybe an hour 
or less.

TASK DETAIL
  https://phabricator.wikimedia.org/T300255

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Michael, Lucas_Werkmeister_WMDE, hoo, LSobanski, Marostegui, Aklapper, 
Ladsgroup, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org