[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Clement_Goubert 
Clement_Goubert closed this task as "Resolved".
Clement_Goubert claimed this task.
Clement_Goubert added a comment.


  I fixed your alert too, which will now alert if p50 on 15 minutes goes over 
10 minutes. We can resolve since you don't appear to have any other alerts on 
cp-jobqueue metrics.

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Clement_Goubert
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  Alright, thanks for making it make sense! Does that mean we can close this?

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread akosiaris 
akosiaris added a comment.


  Fixed the alert too. Took me a bit to figure out how to find it, thanks for 
posting the link in the task.

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: akosiaris
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread akosiaris 
akosiaris added a comment.


  I think I have fixed the graphs now to be correct. They will definitely be 
more correct than previously where they were doing statistically wrong things 
(aggregating aggregates)

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: akosiaris
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread akosiaris 
akosiaris added a comment.


  Sorry about that. For what is worth, we are approaching this piecemeal and 
this is the first instance. There are more changeprop related metrics that are 
wrongly summaries and not histograms, we will ping you before changing the next 
few ones.

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: akosiaris
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Clement_Goubert 
Clement_Goubert added a comment.


  In T341054#8987701 , 
@Lucas_Werkmeister_WMDE wrote:
  
  > In T341054#8987695 , 
@Clement_Goubert wrote:
  >
  >> Ah you have an alert on that metric, sorry :(
  >> We switched the metric to an histogram because aggregation was wrong, the 
job itself is ok.
  >> @akosiaris is fixing the charts.
  >
  > Ah, is that why it changed after a reload? :D
  
  Yep, that was my (bad) attempt at fixing the graph. For what it's worth, 
these graphs aggregated prometheus summaries, which are non-aggregatable, so 
they were wrong anyways.

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Clement_Goubert
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  In T341054#8987695 , 
@Clement_Goubert wrote:
  
  > Ah you have an alert on that metric, sorry :(
  > We switched the metric to an histogram because aggregation was wrong, the 
job itself is ok.
  > @akosiaris is fixing the charts.
  
  Ah, is that why it changed after a reload? :D

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  From an end-user perspective, Wikidata edits on English Wikipedia recent 
changes 

 still seem to arrive as usual, so I don’t think anything’s immediately on fire.

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Clement_Goubert 
Clement_Goubert added a subscriber: akosiaris.
Clement_Goubert added a comment.


  Ah you have an alert on that metric, sorry :(
  We switched the metric to an histogram because aggregation was wrong, the job 
itself is ok.
  @akosiaris is fixing the charts.

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Clement_Goubert
Cc: akosiaris, Michael, Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, 
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  Also, if I reload the Grafana tab, the graph starts to look quite different, 
which is very confusing:
  
  F37127916: image.png 

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a subscriber: Clement_Goubert.
Lucas_Werkmeister_WMDE added a comment.


  The cutoff in the 1h graph seems to be 45 minutes after the cutoff in the 
15min graph, so if we speculate that the cause was 15 minutes before the cutoff 
in the 15min graph, then that would give us 9:42 or so, which would line up 
pretty well with some SAL message about “changeprop-jobqueue” by 
@Clement_Goubert… do you have any idea?

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Clement_Goubert, Aklapper, Lucas_Werkmeister_WMDE, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, Rishacha, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

2023-07-04 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE created this task.
Lucas_Werkmeister_WMDE added projects: Wikidata, wdwb-tech, WMF-JobQueue.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  At 10:04 UTC, the wikidata-monitoring email received an alert about 
“DispatchChanges Normal job backlog time (mean avg, 15min)”:
  
  > [1] Firing
  > Labels
  > alertname = DispatchChanges Normal job backlog time (mean avg, 15min) alert
  > __alert_rule_uid__ = MF0FSjJ4z
  > __contacts__ = "AlertManager","cxserver"
  > datasource_uid = 00026
  > grafana_folder = Wikidata
  > ref_id = A
  > rule_uid = MF0FSjJ4z
  > severity = critical
  > team = wikidata
  > Annotations
  > __alertId__ = 309
  > __dashboardUid__ = TUJ0V-0Zk
  > __orgId__ = 1
  > __panelId__ = 28
  > grafana_state_reason = NoData
  > message = DispatchChanges job backlog is over 10 minutes! Normal values are 
between 0.5s and 1s
  > Source 
  
  According to another email received at 10:24 UTC, the alert was resolved, but 
the job in Grafana 

 still doesn’t look good – the backlog time just cut off:
  
  F37127912: image.png 
  
  We should figure out what’s going on here, and if anything is still broken.

TASK DETAIL
  https://phabricator.wikimedia.org/T341054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Aklapper, Lucas_Werkmeister_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, Rishacha, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Pchelolo, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org