[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-10-28 Thread Gehel 
Gehel closed subtask T288231: Deploy the wdqs streaming updater to production 
as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Gehel
Cc: Lucas_Werkmeister_WMDE, dcausse, Legoktm, Gehel, Aklapper, Invadibot, 
MPhamWMF, maantietaja, CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, 
jijiki, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, 
Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-10-20 Thread dcausse 
dcausse added a comment.


  In T285710#7443613 , 
@Lucas_Werkmeister_WMDE wrote:
  
  > Does that mean the periodic `updateQueryServiceLag` should be removed? 
Currently that’s still running on mwmaint1002.
  
  T221774  was added because WDQS 
could not keep up with the update rate, the streaming updater was made to make 
sure WDQS is no longer the bottleneck here and that edits made by bots 
respecting maxLag are no longer throttled.
  I think this that T221774  is 
indeed no longer needed, my team have plan to have an SLO on the wdqs lag to 
make sure it's always below 10min (T293027 
). Is this enough/equivalent to 
T221774 ? I've filed T293886 
 to continue the discussion.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Lucas_Werkmeister_WMDE, dcausse, Legoktm, Gehel, Aklapper, Invadibot, 
MPhamWMF, maantietaja, CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, 
jijiki, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, 
Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-10-20 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  Does that mean the periodic `updateQueryServiceLag` should be removed? 
Currently that’s still running on mwmaint1002.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, dcausse, Legoktm, Gehel, Aklapper, Invadibot, 
MPhamWMF, maantietaja, CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, 
jijiki, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, 
Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-10-19 Thread dcausse 
dcausse closed this task as "Resolved".
dcausse claimed this task.
dcausse added a comment.


  In T285710#7441835 , 
@Legoktm wrote:
  
  > I think this is resolved now that the streaming updater is in use 
everywhere? 
https://lists.wikimedia.org/hyperkitty/list/wikid...@lists.wikimedia.org/message/NXBOCI3WKTZBB6RB2GYWBBH2BFH3NBT6/
  
  Yes finally! :)

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-10-19 Thread Legoktm 
Legoktm added a comment.


  I think this is resolved now that the streaming updater is in use everywhere? 
https://lists.wikimedia.org/hyperkitty/list/wikid...@lists.wikimedia.org/message/NXBOCI3WKTZBB6RB2GYWBBH2BFH3NBT6/

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Legoktm
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-09-09 Thread Legoktm 
Legoktm removed a parent task: T287539: September 2021 Datacenter switchover 
(codfw -> eqiad).

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Legoktm
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-08-05 Thread dcausse 
dcausse edited subtasks, added: T288231: Deploy the wdqs streaming updater to 
production; removed: T244590: [Epic] Rework the WDQS updater as an event driven 
application.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-08-05 Thread dcausse 
dcausse added a subtask: T244590: [Epic] Rework the WDQS updater as an event 
driven application.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-08-05 Thread Legoktm 
Legoktm added a comment.


  In T285710#7262256 , 
@dcausse wrote:
  
  > Because none of these changes would be trivial I think we prefer to wait 
for the new system to be in place.
  
  Fair enough, thanks for the explanation. Could you add a blocker to this task 
of whichever one is for deploying the new updater?

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Legoktm
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-08-05 Thread dcausse 
dcausse added a comment.


  In T285710#7261551 , 
@Legoktm wrote:
  
  > @Gehel what ends up consuming that value? Can we have it read the primary 
DC from conftool?
  >
  > For now I've documented this as a manual step: 
https://wikitech.wikimedia.org/w/index.php?title=Switch_Datacenter=revision=1920831=1920827
  
  The system is passed a topic name at startup used as a reference for the lag 
(https://gerrit.wikimedia.org/r/plugins/gitiles/wikidata/query/rdf/+/refs/heads/master/tools/src/main/java/org/wikidata/query/rdf/tool/change/KafkaPoller.java#365).
 The updater then only reports the timestamp of this topic to blazegraph which 
is then consumed by a prometheus exporter.
  The way this "timestamp" (lag) is determined should be changed to either:
  
  - determine dynamically what is the "reportingTopic" calling conftool to know 
where eventgate is pushing MW events
  - do what the new updater is doing (T244590 
): compute an avg of the timestamps 
instead
  
  Because none of these changes would be trivial I think we prefer to wait for 
the new system to be in place.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-08-04 Thread Legoktm 
Legoktm added a comment.


  @Gehel what ends up consuming that value? Can we have it read the primary DC 
from conftool?
  
  For now I've documented this as a manual step: 
https://wikitech.wikimedia.org/w/index.php?title=Switch_Datacenter=revision=1920831=1920827

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Legoktm
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-08-04 Thread Legoktm 
Legoktm added a parent task: T287539: September 2021 Datacenter switchover 
(codfw -> eqiad).

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Legoktm
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-06-29 Thread Gehel 
Gehel added a comment.


  It is unlikely that the long term fix (T244590 
) will be in place for the switch 
back. The simple workaround (but far from ideal) is to revert 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/701927/ during the switch 
back. The more complex solution would be to implement better metrics in the 
current updater, but I doubt we will have time to do that before the switch 
back.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-06-28 Thread Addshore 
Addshore added projects: Wikidata, wdwb-tech.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: dcausse, Legoktm, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, RhinosF1, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Krenair, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-06-28 Thread Legoktm 
Legoktm added a project: Datacenter-Switchover.

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Legoktm
Cc: dcausse, Legoktm, Gehel, Aklapper, MPhamWMF, CBogen, RhinosF1, Namenlos314, 
jijiki, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Volans, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Krenair, 
akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285710: WDQS lag detection required manual adjustment during DC switchover

2021-06-28 Thread Legoktm 
Legoktm created this task.
Legoktm triaged this task as "High" priority.
Legoktm added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  During today's DC switchover, the WDQS lag checking/monitoring required 
manual adjustment: https://gerrit.wikimedia.org/r/701927
  
  I'm marking this as high priority because it ended up causing user impact as 
it affected bots that check the lag before editing.
  
  On IRC @gehel said that the long-term fix for this is T244590: [Epic] Rework 
the WDQS updater as an event driven application 
. How long-term are we looking at? 
Will that be in place for the switch back in ~1 month? Or do we also need a 
short term solution?

TASK DETAIL
  https://phabricator.wikimedia.org/T285710

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Legoktm
Cc: dcausse, Legoktm, Gehel, Aklapper, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org