subject:"\[Wikidata\-bugs\] \[Maniphest\] T138208\: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium"

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-05-11 Thread Ladsgroup

Ladsgroup closed this task as "Resolved".
Ladsgroup added a project: DBA.
Ladsgroup claimed this task.
Ladsgroup added a comment.


  One month have passed. We haven't had major issues like we used to (dumps 
running making it impossible to depool any host in s8) so I call this done. We 
can reopen or create new tickets if it happens again in a smaller scale or 
smaller dumps

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, Busfault, Astuthiodit_1, 
karapayneWMDE, Invadibot, Devnull, maantietaja, jannee_e, ItamarWMDE, 
Akuckartz, Iflorez, holger.knust, RhinosF1, Nandana, Lahi, Gq86, 
GoranSMilovanovic, Lunewa, QZanden, LawExplorer, Minhnv-2809, _jensen, 
rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair, 
Hokwelum
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-04-11 Thread ArielGlenn

ArielGlenn added a comment.


  In T138208#7844298 , 
@Ladsgroup wrote:
  
  > It's a bit hard to measure but it's probably fixed.
  
  That would be wonderful if true. Let's leave this open for a while yet just 
in case...

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, Busfault, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, jannee_e, ItamarWMDE, Akuckartz, 
holger.knust, RhinosF1, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331, Hokwelum
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-04-11 Thread Ladsgroup

Ladsgroup added a comment.


  It's a bit hard to measure but it's probably fixed.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, Busfault, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, jannee_e, ItamarWMDE, Akuckartz, 
holger.knust, RhinosF1, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331, Hokwelum
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-04-05 Thread Maintenance_bot

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, Busfault, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, jannee_e, ItamarWMDE, Akuckartz, 
holger.knust, RhinosF1, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331, Fernandobacasegua34, 786, Suran38, Biggs657, Lalamarie69, 
Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, 
CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, 
Maathavan, Neuronton
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-04-05 Thread gerritbot

gerritbot added a comment.


  Change 767477 **merged** by jenkins-bot:
  
  [operations/dumps@master] Add --dbgroupdefault=dump to every major dump run
  
  https://gerrit.wikimedia.org/r/767477

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, Busfault, Fernandobacasegua34, 
Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Hook696, Kent7301, holger.knust, RhinosF1, joker88john, CucyNoiD, 
Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, Lunewa, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Neuronton, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-03-02 Thread Ladsgroup

Ladsgroup added a comment.


  I'm surprised --dbgroupdefault is not set in any dump run, it's a built-in 
system in mediawiki to provide support for cases like this.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, Fernandobacasegua34, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, Juan90264, 
Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, gnosygnu, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-03-02 Thread gerritbot

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, Fernandobacasegua34, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, Juan90264, 
Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, gnosygnu, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-03-02 Thread gerritbot

gerritbot added a comment.


  Change 767477 had a related patch set uploaded (by Ladsgroup; author: Amir 
Sarabadani):
  
  [operations/dumps@master] Add --dbgroupdefault=dump to every major dump run
  
  https://gerrit.wikimedia.org/r/767477

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, 
jannee_e, Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, 
Lunewa, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-03-02 Thread Ladsgroup

Ladsgroup added a comment.


  In T138208#7568960 , 
@ArielGlenn wrote:
  
  > As I feared, fetchText.php calls 
MediaWikiServices::getInstance()->getBlobStore()->getBlob() which gets a db 
replica connection on its own, with no opportunity for us to ask that it be in 
the vslow/dump group. We might be able to use the -dbgroupdefault dump option 
to this script; I will have to do some testing to see if that has any effect 
and what happens when that group is suddenly not available.
  
  I just tested it, it won't break, it simply ignores it.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Kormat, LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, 
daniel, hoo, ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, 
jannee_e, Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, 
Lunewa, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread daniel

daniel added a comment.


  In T138208#7727286 , 
@Ladsgroup wrote:
  
  > At least it can avoid connecting to non-dump hosts.
  
  I thought we did that... maybe @ArielGlenn has an idea.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread ArielGlenn

ArielGlenn added a comment.


  I am aware of and following this discussion but right now, my responsiveness 
on this task will be slow, most of my time needs to go to getting my teammate 
who will be dumps co-maintainer up to speed. Please bear with us.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread Ladsgroup

Ladsgroup added a comment.


  At least it can avoid connecting to non-dump hosts.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread daniel

daniel added a comment.


  In T138208#7727264 , 
@Ladsgroup wrote:
  
  > Possibly but also keeping the connection open? Maybe it needs to buffer, 
close the connection and then compress given that it's cpu intensive and slow?
  
  WikiExporter writes each chunk of xml to an DumpOutput. In the above case, 
that would be a DumpBZip2Output, which is a DumpPipeOutput, which uses 
proc_open to start bzip2 and then writes each chunk to the child process's 
stdin. The output is far too big to buffer in memory. Writing to disk 
uncompressed may be an option, bout would require an order of magnitude more 
disk space. And the wrapper scripts would need to be changed significantly, I 
suppose.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread Ladsgroup

Ladsgroup added a comment.


  Possibly but also keeping the connection open? Maybe it needs to buffer, 
close the connection and then compress given that it's cpu intensive and slow?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread daniel

daniel added a comment.


  In T138208#7727235 , 
@Ladsgroup wrote:
  
  > Most notably the top of the snapshot is not the dumper, it's bzip2. Does it 
keep connection open while compressing?
  
  Isn't it compressing the stream on the fly?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread Ladsgroup

Ladsgroup added a comment.


  I'm fairly certain it does.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread Ladsgroup

Ladsgroup added a comment.


  Most notably the top of the snapshot is not the dumper, it's bzip2. Does it 
keep connection open while compressing?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-02-22 Thread Ladsgroup

Ladsgroup added a comment.


  Today, I was trying to upgrade s8 to bullseye and I can't depool any host, 
they all ended up with lingering connections from snapshot1011.eqiad.wmnet.
  
  For the whole hour in the hosts there, it was just sleep. Looking at 
snapshot1011, the dumper is `/usr/bin/php7.2 
/srv/mediawiki/multiversion/MWScript.php dumpTextPass.php --wiki=wikidatawiki 
--stub=gzip:/mnt/dumpsdata/xmldatadumps/temp/w/wikidatawiki/wikidatawiki-20220220-stub-articles26.xml-p83298894p84798893.gz
 
--prefetch=bzip2:/mnt/dumpsdata/xmldatadumps/public/wikidatawiki/20220201/wikidatawiki-20220201-pages-articles26.xml-p83298894p84798893.bz2
 --report=1000 --spawn=/usr/bin/php7.2 
--output=bzip2:/mnt/dumpsdata/xmldatadumps/public/wikidatawiki/20220220/wikidatawiki-20220220-pages-articles26.xml-p83298894p84798893.bz2.inprog
 --current`

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, karapayneWMDE, Invadibot, maantietaja, jannee_e, 
Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-24 Thread ArielGlenn

ArielGlenn added a comment.


  Thanks. I was pretty careful with my testing for the last fix, making sure 
that in production the patch redirected to a vslow/dump server. But I may have 
overlooked something. :-(

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-24 Thread Marostegui

Marostegui added a comment.


  Next time I will try a different approach to capture them, which I think it 
will give us the exact queries

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-24 Thread Marostegui

Marostegui added a comment.


  I tried but I wasn't able to capture any :(

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-24 Thread ArielGlenn

ArielGlenn added a comment.


  I hate to ask but can we capture any queries?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-24 Thread Marostegui

Marostegui added a comment.


  So I am still seeing a wikiadmin connection from snapshot1011.eqiad.wmnet. to 
a s8 API host (db1172)

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-24 Thread ArielGlenn

ArielGlenn added a comment.


  The above patch was deployed with the train everywhere, so the specific set 
of queries should no longer be directed to non-vslow/dump db servers. If that's 
the cas, we are now back to the harder issue of what to do when a db server is 
depooled, and I think that discussion is happening elsewhere.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-17 Thread daniel

daniel added a comment.

  In T138208#7625170 , 
@Ladsgroup wrote:

  > I think we have different perspectives and it might be because I'm coming 
from SRE? I personally think dumps are actually the environment, not the code.

  I see that point, but I don't want the fix for this issue to be blocked on 
that discussion. Setting the group from an env variable would be a new 
approach. Also, forcing the group (rather than changing the default) would be 
new. I suggest opening a separate ticket for that idea.

  > Beside the maint. script itself, they basically share the same code as 
Special:Export and that is being used from the appservers.

  It shared code, but the access pattern is vastly different.

  > Currently basically all db queries directly called from wikibase has a 
dedicated group such as "from-client" or "from-repo". We need something to 
override that for the sake of reliability.

  Hm... this is starting to sound like we want consider a set of hints when 
choosing the replica, rather than specifying a group. Interesting idea.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-17 Thread Maintenance_bot

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331, 786, Suran38, Biggs657, Lalamarie69, Juan90264, Alter-paule, 
Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, 
Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-17 Thread ReleaseTaggerBot

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.18; 2022-01-17).

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ReleaseTaggerBot
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-17 Thread gerritbot

gerritbot added a comment.


  Change 747455 **merged** by jenkins-bot:
  
  [mediawiki/core@master] try to use 'dump' group for db connections for dumps 
of page content
  
  https://gerrit.wikimedia.org/r/747455

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-17 Thread ArielGlenn

ArielGlenn added a comment.


  The patch at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/747455/ is 
tested and ready to go, and in line with the way existing dumps scripts work. 
So I'd like to go ahead with it.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-16 Thread Ladsgroup

Ladsgroup added a comment.


  I think we have different perspectives and it might be because I'm coming 
from SRE? I personally think dumps are actually the environment, not the code. 
Beside the maint. script itself, they basically share the same code as 
Special:Export and that is being used from the appservers. And I think given 
that dumps has a dedicated group in dbs (similar to api), it feels more 
environment-y than code-y.  (And we are removing code-y groups such as 
watchlist from dbs). The other reason is that for third parties, they can be 
using dump scripts but not needing a dedicated dumps group. And last but not 
least, that global variable doesn't override when a method asks for a different 
group. e.g. Currently basically all db queries directly called from wikibase 
has a dedicated group such as "from-client" or "from-repo". We need something 
to override that for the sake of reliability.
  
  I would be open to any idea that would be easy to implement though.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-13 Thread daniel

daniel added a comment.

  In T138208#7612881 , 
@Ladsgroup wrote:

  > Thanks, that was the missing piece. My suggestion would be to set an 
environmental variable in calling mw scripts (if it's not set already). phpunit 
does a similar thing. And in LB code in mw when trying to get a connection, it 
should check the env variable and override groups. That would be the cleanest 
way if you ask me.
  >
  > We still have the problem of config reload, how long each stream job 
usually takes?

  An environment variable would work, but I think I prefer the idea of setting 
the group per script. Wikidata's DumpEntities does this, and it makes more 
sense to me semantically: defaulting to a specific server group is specific to 
the script's task, not to the environment it runs in.

  Having a way to tell getBlob to use a different db group, as I suggested 
earlier, would be nice, but it's rather brittle. We will have to remember to 
allow such a flag to be passed into all storage layer services. That doesn't 
seem great.

  There's a caveat wrt settings $wgDBDefaultGroup in finalSetup(): There is no 
guarantee that whatever service is going to read that variable hasn't already 
been instantiated. Specifically note that MWLBFactory will copy the value of 
DBDefaultGroup, and any modification of the global will have no effect after 
that.

  Mid-term, we will need a canonical way for maintenance scripts to set 
configuration that does not involve global variables. This has to happen after 
all configuration has been loaded, but before any services have been 
constructed. Wikidata's DumpEntities uses the MediaWikiServices hook to do 
that, but it would be nice to make this easier for maintenance scripts. Perhaps 
finalSetup should just always run in a MediaWikiServices hook? See T275334 
 and T271574 
 for related discussion.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-11 Thread Ladsgroup

Ladsgroup added a comment.


  Thanks, that was the missing piece. My suggestion would be to set an 
environmental variable in calling mw scripts (if it's not set already). phpunit 
does a similar thing. And in LB code in mw when trying to get a connection, it 
should check the env variable and override groups. That would be the cleanest 
way if you ask me.
  
  We still have the problem of config reload, how long each stream job usually 
takes?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-10 Thread ArielGlenn

ArielGlenn added a comment.


  There is a complicated set of python scripts that coordinate the dump jobs 
for each wiki during the two monthly runs. 
https://wikitech.wikimedia.org/wiki/Dumps/Current_Architecture gives an 
overview. 
https://www.mediawiki.org/wiki/SQL/XML_Dumps#Becoming_a_dumps_co-maintainer 
gives rather a lot more. In general for testing you will run the python 
worker.py script, supplying it with the config file, the job name, the run date 
and the wiki; we test in deployment-prep, although I am working on a docker 
container testbed.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-10 Thread Ladsgroup

Ladsgroup added a comment.


  More on operations part of it, how they are being ran? Is it cron/systemd 
timer? if so where? What is the maint script or bash script that is being ran, 
etc.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-10 Thread ArielGlenn

ArielGlenn added a comment.

  In T138208#7611718 , 
@Ladsgroup wrote:

  > In T138208#7611712 , 
@ArielGlenn wrote:
  >
  >> Not yet; I need to talk with someone more knowledgeable than me about 
whether this approach is reasonable, before moving forward. I'll bring it up at 
our next meeting (tomorrow).
  >
  > Can I know how dumpers work? Any link to documentation would be 
appreciated. I need it to understand this patch and also finding a way for 
T298485 

  I don't know of any documentation specifically for the MW maintenance scripts 
for dumps or the modules used for import/export. There are genreal Manual pages 
for importing and exporting (maintained by volunteers I think) but I don't 
think they have the level of detail you are looking for. I have plenty of 
documentation for the python scripts, the formats, the content, and the various 
servers and how they are set up. But I guess that won't be so helpful here. 
Should we meet? Should I try to write something? If so, how in depth does it 
need to be?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-10 Thread Ladsgroup

Ladsgroup added a comment.


  In T138208#7611712 , 
@ArielGlenn wrote:
  
  > Not yet; I need to talk with someone more knowledgeable than me about 
whether this approach is reasonable, before moving forward. I'll bring it up at 
our next meeting (tomorrow).
  
  Can I know how dumpers work? Any link to documentation would be appreciated. 
I need it to understand this patch and also finding a way for T298485 


TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-10 Thread ArielGlenn

ArielGlenn added a comment.


  In T138208#7611708 , 
@Marostegui wrote:
  
  > In T138208#7571559 , 
@gerritbot wrote:
  >
  >> Change 747455 had a related patch set uploaded (by ArielGlenn; author: 
ArielGlenn):
  >>
  >> [mediawiki/core@master] try to use 'dump' group for db connections for 
dumps of page content
  >>
  >> https://gerrit.wikimedia.org/r/747455
  >
  > Any ETA on when this will be merged? Thanks!
  
  Not yet; I need to talk with someone more knowledgeable than me about whether 
this approach is reasonable, before moving forward. I'll bring it up at our 
next meeting (tomorrow).

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2022-01-10 Thread Marostegui

Marostegui added a comment.


  In T138208#7571559 , 
@gerritbot wrote:
  
  > Change 747455 had a related patch set uploaded (by ArielGlenn; author: 
ArielGlenn):
  >
  > [mediawiki/core@master] try to use 'dump' group for db connections for 
dumps of page content
  >
  > https://gerrit.wikimedia.org/r/747455
  
  Any ETA on when this will be merged? Thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-15 Thread gerritbot

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
maantietaja, Juan90264, Alter-paule, jannee_e, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-15 Thread gerritbot

gerritbot added a comment.


  Change 747455 had a related patch set uploaded (by ArielGlenn; author: 
ArielGlenn):
  
  [mediawiki/core@master] try to use 'dump' group for db connections for dumps 
of page content
  
  https://gerrit.wikimedia.org/r/747455

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-14 Thread ArielGlenn

ArielGlenn added a comment.


  Thanks for this thought, Daniel. I think it's better if I can pass the 
dbgroupdefault parameter to the maintenance script itself, instead of hacking 
something into getBlob(). But I do need to check if that's going to work ok. 
The longer term fix you mentioned, is there a task for that, so I can follow 
along?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-14 Thread daniel

daniel added a comment.


  In T138208#7568960 , 
@ArielGlenn wrote:
  
  > As I feared, fetchText.php calls 
MediaWikiServices::getInstance()->getBlobStore()->getBlob() which gets a db 
replica connection on its own, with no opportunity for us to ask that it be in 
the vslow/dump group.
  
  This can be fixed easily enough. We can just add a parameter to getBlob(), 
and pass it on to lower layers as appropriate. I'd be happy to work on that 
with you if you like.
  
  The real solution would be not not hit the text table at all any more, but 
move the external store reference into content_address. We would cut out the 
middle-table, so to speak. There is no good reason to use the text table when 
external store is enabled, the only thing that is holding us back is the fact 
that the migration takes some effort.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-14 Thread Marostegui

Marostegui added a comment.


  Thanks for looking into this issue :-)

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-14 Thread ArielGlenn

ArielGlenn added a comment.


  As I feared, fetchText.php calls 
MediaWikiServices::getInstance()->getBlobStore()->getBlob() which gets a db 
replica connection on its own, with no opportunity for us to ask that it be in 
the vslow/dump group. We might be able to use the -dbgroupdefault dump option 
to this script; I will have to do some testing to see if that has any effect 
and what happens when that group is suddenly not available.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-13 Thread ArielGlenn

ArielGlenn added a comment.


  The above is happening from pages-meta-history dumps, and I will look into it 
later today. The snapshot1008 (wikidata entity) dumps will be harder.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-13 Thread Marostegui

Marostegui added a comment.


  Following my chat with @ArielGlenn on IRC, I got this query coming from 
snapshot1011
  
2690453130wikiadmin10.64.0.156:38912wikidatawikiQuery0  
  StatisticsSELECT /* MediaWiki\\Storage\\SqlBlobStore::fetchBlobs 
dumpsgen@snapsh... */  old_id,old_text,old_flags  FROM `text`WHERE old_id = 
15429306430.000

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-13 Thread ArielGlenn

ArielGlenn added a comment.


  The reason only those two snapshot hosts are involved is undoubtedly because 
dumps on the others have finished for this run.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: LSobanski, Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, 
ArielGlenn, jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, 
holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-13 Thread Marostegui

Marostegui added a comment.


  Now that I look into it, the `wikiadmin` user is connected to all the pooled 
hosts, regardless of their config group. Connections are from snapshost1008 and 
snapshost1011
  This would make impossible to run an automated schema change on any of those 
wikidata hosts unless those processes have finished.
  
  I am going to manually run the schema change for now, but I would appreciate 
if some research could be done on how to overcome this.
  
  Thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, 
jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, holger.knust, 
Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-12-13 Thread Marostegui

Marostegui added a subscriber: Ladsgroup.
Marostegui added a comment.


  This keeps handling and it is painful if we want to run automated schema 
changes (T288235 ). I have tried to 
deploy a change on s8 without manual intervention but it got stuck on:
  db
  db1114
  db1126
  
  All of the them had wikiadmin user connecting coming from snapshot hosts 
(snapshot1011) and that prevents the script to continue as it waits for all the 
connections to get drained. 
  None of those hosts are part of the vslow,dump group. They are part of main 
and api traffic groups.
  Could this be given some priority?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: Ladsgroup, Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, 
jcrespo, Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, holger.knust, 
Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-09-21 Thread Addshore

Addshore closed subtask T138381: Allow DB group used by ChangeDispatcher to be 
configured as Declined.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, 
Zppix, Invadibot, maantietaja, jannee_e, Akuckartz, holger.knust, Nandana, 
Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-02-09 Thread ArielGlenn

ArielGlenn added a comment.


  mysql.php, used for wikidata entity dumps, does not apparently correctly 
handle the --group flag. it's unclear to me what it does do, I need to check 
into this sometime later. The queries run by it are extremely short so the 
impact is minimal, but it still needs to be checked.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, 
Zppix, jannee_e, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-02-08 Thread ArielGlenn

ArielGlenn added a comment.


  In T138208#6811418 , 
@Addshore wrote:
  
  > In T138208#6809784 , 
@ArielGlenn wrote:
  >
  >> This is because the maintenance scripts that do "small" page ranges take 
several hours to complete. I will keep this in mind for when we can go to 
multiple bz2 streams in the page content history dumps; I'll be able to dump 
much smaller ranges then and concat them together. The other thing I should do 
is check how often we respawn fetchText; that is something I might be able to 
change sooner rather than later.
  >
  > From the sounds of things I can leave this ticket on your plate then 
@ArielGlenn ? :)
  
  Sadly, yes :-P

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, 
Zppix, jannee_e, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-02-08 Thread Addshore

Addshore moved this task from Discussion to External Realm on the 
wdwb-tech-focus board.
Addshore added a comment.


  In T138208#6809784 , 
@ArielGlenn wrote:
  
  > This is because the maintenance scripts that do "small" page ranges take 
several hours to complete. I will keep this in mind for when we can go to 
multiple bz2 streams in the page content history dumps; I'll be able to dump 
much smaller ranges then and concat them together. The other thing I should do 
is check how often we respawn fetchText; that is something I might be able to 
change sooner rather than later.
  
  From the sounds of things I can leave this ticket on your plate then 
@ArielGlenn ? :)

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

WORKBOARD
  https://phabricator.wikimedia.org/project/board/3532/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, 
Zppix, jannee_e, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-02-08 Thread ArielGlenn

ArielGlenn added a comment.


  This is because the maintenance scripts that do "small" page ranges take 
several hours to complete. I will keep this in mind for when we can go to 
multiple bz2 streams in the page content history dumps; I'll be able to dump 
much smaller ranges then and concat them together. The other thing I should do 
is check how often we respawn fetchText; that is something I might be able to 
change sooner rather than later.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, 
Zppix, jannee_e, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-02-08 Thread Marostegui

Marostegui added a comment.


  I depooled today db (https://phabricator.wikimedia.org/P14229) which is 
not on the vslow,dump group and I still see two things:
  
  - The host is being accessed by `wikiadmin` from `snapshot1006`:
  - The host keeps being accessed even though it's been depooled for more than 
1 hour:
  
root@cumin1001:/home/marostegui# mysql.py -hdb -e "show processlist" | 
grep wikiadmin
2017398773  wikiadmin   10.64.32.149:60598  wikidatawikiSleep   
803 NULL0.000
2018035580  wikiadmin   10.64.32.149:60960  wikidatawikiSleep   
2   NULL0.000
2018117400  wikiadmin   10.64.32.149:32822  wikidatawikiSleep   
19  NULL0.000
2018253068  wikiadmin   10.64.32.149:33008  wikidatawikiSleep   
1   NULL0.000
2018279747  wikiadmin   10.64.32.149:33028  wikidatawikiSleep   
403 NULL0.000
2018352112  wikiadmin   10.64.32.149:33082  wikidatawikiSleep   
9   NULL0.000
2018442606  wikiadmin   10.64.32.149:33152  wikidatawikiSleep   
318 NULL0.000
2019017411  wikiadmin   10.64.32.149:33300  wikidatawikiSleep   
767 NULL0.000
root@cumin1001:/home/marostegui# host 10.64.32.149
149.32.64.10.in-addr.arpa domain name pointer snapshot1006.eqiad.wmnet.
  
  
  
root@snapshot1006:~# host db
db.eqiad.wmnet has address 10.64.0.128
root@snapshot1006:~# netstat -putan | grep "10.64.0.128"
tcp0  0 10.64.32.149:60598  10.64.0.128:3306
ESTABLISHED 25933/php7.2
tcp0  0 10.64.32.149:33300  10.64.0.128:3306
ESTABLISHED 25711/php7.2
tcp0  0 10.64.32.149:33028  10.64.0.128:3306
ESTABLISHED 25704/php7.2
tcp0  0 10.64.32.149:60960  10.64.0.128:3306
ESTABLISHED 29537/php7.2
tcp0  0 10.64.32.149:33152  10.64.0.128:3306
ESTABLISHED 30709/php7.2
tcp0  0 10.64.32.149:33082  10.64.0.128:3306
ESTABLISHED 30461/php7.2
tcp0  0 10.64.32.149:33008  10.64.0.128:3306
ESTABLISHED 30226/php7.2
tcp0  0 10.64.32.149:32822  10.64.0.128:3306
ESTABLISHED 29707/php7.2
root@snapshot1006:~# ps aux | grep 25933
dumpsgen 25933  0.0  0.0 457784 63076 ?S06:38   0:03 
/usr/bin/php7.2 /srv/mediawiki/php-1.36.0-wmf.27/../multiversion/MWScript.php 
fetchText.php --wiki wikidatawiki
root@snapshot1006:~# ps aux | grep 30709
dumpsgen 30709  0.0  0.1 461880 66992 ?S06:58   0:02 
/usr/bin/php7.2 /srv/mediawiki/php-1.36.0-wmf.27/../multiversion/MWScript.php 
fetchText.php --wiki wikidatawiki

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Marostegui
Cc: Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, 
Zppix, jannee_e, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-01-26 Thread jcrespo

jcrespo added a subscriber: Marostegui.
jcrespo added a comment.


  Handover to @Marostegui for him to comment, as he will be the person to know 
if this continues happening or now.

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, 
Zppix, jannee_e, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium

2021-01-22 Thread Addshore

Addshore moved this task from Backlog to Discussion on the wdwb-tech-focus 
board.
Addshore added a comment.


  Trying to decode if there are any further concrete actionables here and who 
needs to action them?

TASK DETAIL
  https://phabricator.wikimedia.org/T138208

WORKBOARD
  https://phabricator.wikimedia.org/project/board/3532/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, 
jannee_e, Akuckartz, Nandana, Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, 
Lunewa, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

57 matches

Mail list logo