[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Ladsgroup]

Ladsgroup added a comment.


  In T243701#6152282 , 
@Gehel wrote:
  
  > I'm wondering if exposing both MySQL lag and WDQS lag as a single value is 
wise. Those 2 lags represent different things and clients should be free to 
behave differently depending on their needs.
  
  I'm not 100% sure if wise or not but I want to point out that bots use this 
as "one number to model the stress on the infra" so they can back off if it's 
more then a certain threshold. It already gets the size of jobqueue and divides 
it by 300 and also get the dispatching lag (to client wikis) and divide it to 
60 (IIRC) and picks the highest number. So in the sense of "all models are 
wrong but some are useful", it's a pretty polymorphic model already and I don't 
think the issue is that.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: PeterBowman, matej_suchanek, Masti, THE_IT, Jony, Lydia_Pintscher, 
Vojtech.dostal, William_Avery, valhallasw, SilentSpike, Bugreporter, Demian, 
Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, JeanFred, 
Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, Albertvillanovadelmoral, Xqt, 
Lucas_Werkmeister_WMDE, Addshore, jcrespo, Dvorapa, Aklapper, Strainu, CBogen, 
darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, Manybubbles, 
faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Gehel]

Gehel added a comment.


  In T243701#5985617 , 
@Ladsgroup wrote:
  
  > I think this would be a decision by @Lydia_Pintscher plus we need more info 
from Search platform to know if the issue of WDQS is going to get fixed soon or 
not (or any ETA)
  
  From the Search Platform side, we're still a few month from a robust solution 
to WDQS updates. And even with our proposed streaming updater, we do expect lag 
to climb on occasions, that's the nature of asynchronous systems. We need a 
long term robust way to deal with WDQS lag.
  
  I'm wondering if exposing both MySQL lag and WDQS lag as a single value is 
wise. Those 2 lags represent different things and clients should be free to 
behave differently depending on their needs.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: PeterBowman, matej_suchanek, Masti, THE_IT, Jony, Lydia_Pintscher, 
Vojtech.dostal, William_Avery, valhallasw, SilentSpike, Bugreporter, Demian, 
Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, JeanFred, 
Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, Albertvillanovadelmoral, Xqt, 
Lucas_Werkmeister_WMDE, Addshore, jcrespo, Dvorapa, Aklapper, Strainu, CBogen, 
darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, Manybubbles, 
faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Dvorapa]

Dvorapa added a comment.


  Is this some random spike? :D
  F31763816: obrazek.png 

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Masti, THE_IT, Jony, Lydia_Pintscher, Vojtech.dostal, William_Avery, 
valhallasw, SilentSpike, Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, 
Ladsgroup, Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, faidon, Mbch331, Rxy, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Dvorapa]

Dvorapa added a comment.


  Is there any short-term temporary solution until RFC gets done? Perhaps 
hardcode maxlag=4? Or temporarily revert WDQS inclusion into maxlag?

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Vojtech.dostal, William_Avery, valhallasw, SilentSpike, Bugreporter, 
Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, 
JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, 
Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, jcrespo, 
Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Ladsgroup]

Ladsgroup added a comment.


  In T243701#5973374 , 
@Dvorapa wrote:
  
  > Any news? From possible solutions like T238751 
, T240442 
, T245144 
 and @Ladsgroup's T247459 
?
  > Last 48 hours: 
https://grafana.wikimedia.org/d/00156/wikidata-dispatch?orgId=1&fullscreen&panelId=22&from=now-2d&to=now
  
  The RFC takes some time to get approved and implemented, that should be the 
mid-term plan

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Vojtech.dostal, William_Avery, valhallasw, SilentSpike, Bugreporter, 
Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, 
JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, 
Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, jcrespo, 
Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Vojtech.dostal]

Vojtech.dostal added a comment.


  This is also extremely important for OpenRefine users. We cannot push our 
reconciliations into Wikidata if the lag > maxlag.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Vojtech.dostal
Cc: Vojtech.dostal, William_Avery, valhallasw, SilentSpike, Bugreporter, 
Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, 
JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, 
Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, jcrespo, 
Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Bugreporter]

Bugreporter added a comment.


  By default pywikibot will do one edit every 10 seconds (or longer if you use 
parallel processes). This may be overrided by setting -pt:1 (or 0) and some 
bots runs under it. This task will unconditionally throttle it when lag is high 
(but they can still edit in a high speed when lag is low).

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Strainu]

Strainu added a comment.


  In T243701#5884801 , 
@Ladsgroup wrote:
  
  > I have been thinking about this and I think I have a suggestion. bots and 
tools, should respect maxlag **before** it reaches the threshold. We need to 
return the value of the maxlag in the API as well but tool developers should 
check the maxlag, even if it's 3 or 2 seconds, then wait for that amount and 
make the next edit after that (or just sleep exactly after the edit is made). 
This way they slow down before "they reach the threshold and then have to back 
off the let go back to normal and then they start the flood of edits again and 
so on and so forth."
  
  I'm not sure I follow, pywikibot at least has a write throttle which is 10s 
by default and the issue is still happening. Also, your proposal would only 
work if the clients would apply lagging only for write operations, not read 
ops, otherwise running bots would take forever.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Strainu
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Bugreporter]

Bugreporter added a comment.


  Currently Widar use a policy of sleeping 3x+1 second if the lag (x second) is 
higher than one second. PetScan run batches five in parallel, so for a lag of 
10 seconds PetScan make 5 edits every 31 second.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Ladsgroup]

Ladsgroup added a comment.


  In T243701#5880988 , 
@Bugreporter wrote:
  
  > The only way to resolve the issue is increase the rate of edit Query 
Updater can handle, or reduce the number of triple changes each edit.
  
  Yes and no. Yes, WDQS update is our current bottleneck in the infrastructure 
on handling the sheer amount of edits on Wikidata (which is a good thing) but 
if we fix this bottleneck and make scaling possible, we will sooner or later 
hit another bottleneck and maxlag is a not good way to help. We need to fix how 
we handle maxlag.
  
  I have been thinking about this and I think I have a suggestion. bots and 
tools, should respect maxlag **before** it reaches the threshold. We need to 
return the value of the maxlag in the API as well but tool developers should 
check the maxlag, even if it's 3 or 2 seconds, then wait for that amount and 
make the next edit after that (or just sleep exactly after the edit is made). 
This way they slow down before "they reach the threshold and then have to back 
off the let go back to normal and then they start the flood of edits again and 
so on and so forth."
  
  What do you think?

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Bugreporter]

Bugreporter added a comment.


  The only way to resolve the issue is increase the rate of edit Query Updater 
can handle.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Lea_Lacroix_WMDE]

Lea_Lacroix_WMDE added a comment.


  Thanks all for your feedback. Since the change we perform didn't have the 
expected results, we're going to revert it today, and keep looking for 
sustainable solutions.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lea_Lacroix_WMDE
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[ArthurPSmith]

ArthurPSmith added a comment.


  @Bugreporter
  
  > I think increase the factor will not make thing better, it only increase 
the oscillating period
  
  Yes that does seem to have happened - instead of a roughly 20 minute cycle, 
we now have about a 1-hour cycle.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArthurPSmith
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Bugreporter]

Bugreporter added a comment.


  If the rate of edit Query Updater can handle is a constant, changing the 
factor will not affect the average edit rate, nor the proportion of time the 
lag under a specific maxlag (assuming the rate of edit over maxlag and under 
maxlag are constants independent of the actual maxlag).

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Bugreporter]

Bugreporter added a comment.


  I think increase the factor will not make thing better, it only increase the 
oscillating period. It even make query service worse (more lagged).
  
  See 
https://grafana.wikimedia.org/d/00170/wikidata-edits?orgId=1&from=1581429584959&to=1581542357438

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Bugreporter, Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, 
Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, 
ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, 
jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, 
Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Lea_Lacroix_WMDE]

Lea_Lacroix_WMDE added a comment.


  We just increased the factor to 180 
. If you are running a bot and still 
encountering issues frequently, please let us know.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lea_Lacroix_WMDE
Cc: Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, 
JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, 
Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, jcrespo, 
Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Lea_Lacroix_WMDE]

Lea_Lacroix_WMDE added a comment.


  Hello all,
  Here are some news: we are going to try and increase the maxlag connected to 
the WDQS to **15min **, to see how 
it goes and if most of the problems you encounter still occurs. This change 
should be applied later this week.
  Also, as it seems that the issue with **accessing data** comes from 
Pywikibot, we suggested the developers 
 to remove this limit.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lea_Lacroix_WMDE
Cc: Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, 
JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, 
Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, jcrespo, 
Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Ladsgroup]

Ladsgroup added a comment.


  Yes, I think `wgWikidataOrgQueryServiceMaxLagFactor` should be way higher. 
Something like 120 or 300.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, 
JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, 
Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, jcrespo, 
Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.


  > Either the lag in WDQS needs to be fixed, or we need to introduce some 
scaling factor in Wikibase so that lag is usually under 5s like we have for the 
job queue.
  
  There //is// a scaling factor, the actual threshold is 5 minutes I believe 
(`wgWikidataOrgQueryServiceMaxLagFactor` is 60 in mediawiki-config.git).

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Demian, Legoktm, Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, 
JeanFred, Pasleim, Gehel, Lea_Lacroix_WMDE, ArthurPSmith, 
Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, Addshore, jcrespo, 
Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Vali.matei, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, 
Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[ArthurPSmith]

ArthurPSmith added a comment.


  In T243701#5855439 , 
@ArielGlenn wrote:
  
  > In T243701#5855352 , 
@Lea_Lacroix_WMDE wrote:
  >
  >> Over the past weeks, we noticed a huge increase of content in Wikidata. 
Maybe that's something worth looking at?
  >
  > Wikidata content is growing at a fast and steady pace and has been for a 
few years now. For the last few months it's been expanding at a rate of around 
3,500,000 new pages per month. So that seems unlikely to be connected.
  
  That rate is a lot higher than it was for the first 7 months of 2019, at 
close to or less than 1 million/month, so it could be related. But given the 
existing size of Wikidata, I'd call it a moderate increase, not a "huge 
increase", unless it's much bigger in some other metric than just number of 
items?
  
  On the question of GET:
  
  > In T243701#5834792 , 
@Lucas_Werkmeister_WMDE wrote:
  > I wonder if it would make sense to ignore query service lag on GET 
requests? Those requests shouldn’t put any kind of load on the query service, 
after all.
  
  Is the idea here to split the "lag" parameter into separate ones for GET's 
and edit's? That makes a lot of sense to me...

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArthurPSmith
Cc: Aschroet, ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, JeanFred, Pasleim, 
Gehel, Lea_Lacroix_WMDE, ArthurPSmith, Albertvillanovadelmoral, Xqt, 
Lucas_Werkmeister_WMDE, Addshore, jcrespo, Dvorapa, Aklapper, Strainu, 
darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[ArielGlenn]

ArielGlenn added a comment.


  In T243701#5855352 , 
@Lea_Lacroix_WMDE wrote:
  
  > Over the past weeks, we noticed a huge increase of content in Wikidata. 
Maybe that's something worth looking at?
  
  Wikidata content is growing at a fast and steady pace and has been for a few 
years now. For the last few months it's been expanding at a rate of around 
3,500,000 new pages per month. So that seems unlikely to be connected.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn
Cc: ArielGlenn, Ladsgroup, Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, 
Lea_Lacroix_WMDE, ArthurPSmith, Albertvillanovadelmoral, Xqt, 
Lucas_Werkmeister_WMDE, Addshore, jcrespo, Dvorapa, Aklapper, Strainu, 
darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Lea_Lacroix_WMDE]

Lea_Lacroix_WMDE added a comment.


  Over the past weeks, we noticed a huge increase of content in Wikidata. Maybe 
that's something worth looking at?

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lea_Lacroix_WMDE
Cc: Ladsgroup, Alicia_Fagerving_WMSE, JeanFred, Pasleim, Gehel, 
Lea_Lacroix_WMDE, ArthurPSmith, Albertvillanovadelmoral, Xqt, 
Lucas_Werkmeister_WMDE, Addshore, jcrespo, Dvorapa, Aklapper, Strainu, 
darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[ArthurPSmith]

ArthurPSmith added a comment.


  @Addshore and others - the problem has deteriorated since Saturday - see this 
discussion on Wikidata: 
https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team/Query_Service_and_search#WDQS_lag

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArthurPSmith
Cc: ArthurPSmith, Albertvillanovadelmoral, Xqt, Lucas_Werkmeister_WMDE, 
Addshore, jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, 
Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, 
Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, 
LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, rosalieper, Scott_WUaS, 
Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, Manybubbles, 
Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Strainu]

Strainu added a comment.


  In T243701#5834731 , 
@Addshore wrote:
  
  > Should tests be using the production api and site?
  > What are these tests doing?
  
  Please don't limit this to just tests. It is affecting normal, 
community-approved bot runs as well.
  
  In T243701#5834698 , 
@jcrespo wrote:
  
  > I would like to know if WDQ team has promised <5s lag, as I would be 
surprised, given that canonical data storage of all wikis (mariadb), including 
Wikidata, is only considered lagged starting at 5-10 seconds of delay, and that 
would be a lower bound (blocker) before indexing and postprocessing, not 
including downtime.
  
  As others have mentioned, Pyikibot struggles with the MediaWiki lag,  which 
apparently is linked to the WDQ one.  However, in the Grafana link I provided 
there is a horizontal line at 5s, which suggests that 5s should be the upper 
limit for that particular lag.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Strainu
Cc: Xqt, Lucas_Werkmeister_WMDE, Addshore, jcrespo, Dvorapa, Aklapper, Strainu, 
darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Addshore]

Addshore added a comment.


  Its starting to feel like we should just create a better mechanism for this 
instead of using maxlag.
  Per what I said in T240442#5815397 
 it might be nice to have 
wikibase / mediawiki tell a client how long to perhaps wait for between 
requests rather than indicating this maxlag and then process being to stop when 
it reaches 5.
  This would allow the software to make the decision for reads vs writes etc 
and also slowly decrease rate rather than the very big jump we currently have 
at 5 maxlag.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Lucas_Werkmeister_WMDE, Addshore, jcrespo, Dvorapa, Aklapper, Strainu, 
darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.


  I wonder if it would make sense to ignore query service lag on GET requests? 
Those requests shouldn’t put any kind of load on the query service, after all.
  
  (On the other hand, that might make certain problems annoyingly difficult to 
debug, if the lag is sometimes there but not always.)

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, Addshore, jcrespo, Dvorapa, Aklapper, Strainu, 
darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Dvorapa]

Dvorapa added a comment.


  > For the specific issue you are facing, I may be suggest to review SLA 
expectations about the api (any of it)- and timing out and erroring quickly 
rather than waiting in case of lag for non-interactive tests.
  
  Yeah, there are two options: either make them fail earlier, or make them not 
erroring the whole test suite because of their inresponsiveness. Or both...

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Addshore, jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, 
Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, 
Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Dvorapa]

Dvorapa added a comment.


  > What are these tests doing?
  
  They are obviously testing Pywikibot functions against several wikiproject 
APIs. WD API is usually asked for simple things like some maintenance 
category/template fro some language. Which takes 300s+ to response last weeks 
randomly and today the whole day.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Addshore, jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, 
Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, 
Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[jcrespo]

jcrespo added a comment.


  > API maxlag has been repeatedly declared to always be <5s by several people 
in the past
  
  That would sound about right for Mediawiki API (5-10 seconds) ...
  
  > If WDQS lag is high, API maxlag is also high
  
  ... but this sounds worrying if true- I would wait for wikidata experts to 
comment on that, which is new to me. That dependency does indeed look like a 
real problem to me, as I don't think WDQS can keep up with the same promise at 
the moment based on my casual observation and understanding of the 
architecture. Thanks for bringing this up, as this has deeper infrastructure 
implications due to service dependencies. While I understand the need of 
"slowing bot edits" in case of lag, I can see a problem making MW api and WDQS 
equivalent in SLA at the moment.
  
  > Where have you found WDQS lag promises?
  
  I searched on mediawiki.org and Phabricator to see relevant discussions about 
expected uptime and latency & lag.
  
  For the specific issue you are facing, I may be suggest to review SLA 
expectations about the api (any of it)- and timing out and erroring quickly 
rather than waiting in case of lag for non-interactive tests.

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: Addshore, jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, 
Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, 
Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)

[Dvorapa]

Dvorapa added a comment.


  (please keep in mind I don't understand the differences, all I understand is 
that API lag is outrageously high, probably because WDQS lag is high as those 
two were connected 2 months ago and since then all our tests are sometimes 
failing with 300s timeouts or 50min timeouts, which are two default timeouts 
for Pywikibot tests)

TASK DETAIL
  https://phabricator.wikimedia.org/T243701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Addshore, jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, 
Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, 
Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, 
rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, 
Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, 
Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs