[Wikidata-bugs] [Maniphest] T348443: Investigate incident runbook formats

2023-10-18 Thread Michael 
Michael moved this task from Ready for Peer Review to Done on the Wikidata Dev 
Team (Wikidata.org Slice) board.
Michael closed this task as "Resolved".
Michael claimed this task.
Michael added a comment.


  Thanks! I think with this input we can move forward with the parent task.

TASK DETAIL
  https://phabricator.wikimedia.org/T348443

WORKBOARD
  https://phabricator.wikimedia.org/project/board/6751/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Lucas_Werkmeister_WMDE, hoo, Aklapper, Michael, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T348443: Investigate incident runbook formats

2023-10-18 Thread hoo 
hoo added a comment.


  In T348443#9243403 , 
@Michael wrote:
  
  > So, I would suggest the following rough structure/sections for the Runbook:
  >  […]
  > Thoughts?
  
  Sounds good to me!
  
  In T348443#9258306 , 
@Michael wrote:
  
  > […]
  > Indeed, I did not find many runbooks about alerts specifically. Having the 
current version of that runbook is good, though it looks like not much has 
changed in the structure at first glance.
  > […]
  
  Some of these seem to be named `Monitoring/…` (e.g. Monitoring/atftpd 
),  but these are mostly 
covering general auxiliary services. Given we have a different (much more 
project specific) scope than these alerts, I still think I like 
`WMDE/Wikidata/…` better.

TASK DETAIL
  https://phabricator.wikimedia.org/T348443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo
Cc: Lucas_Werkmeister_WMDE, hoo, Aklapper, Michael, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T348443: Investigate incident runbook formats

2023-10-17 Thread Michael 
Michael added a comment.


  Thanks!
  
  In T348443#9258124 , 
@Lucas_Werkmeister_WMDE wrote:
  
  >> Looking at the naming of the pages in that category, I would suggest 
something like "Wikidata/Runbooks/Change dispatching" or 
"Wikidata/Runbooks/Change dispatching/Alert" as the page name.
  >
  > Given that we already have WMDE/Wikidata 
 I’d use that prefix (I’m 
not sure if you meant to imply that or not). Otherwise that sounds good to me. 
(I’d lean towards the latter option, with “alert” in the title.)
  
  In effect, I meant to use the WMDE/Wikidata prefix. I think I mainly spent 
time thinking about the latter part of the title and didn't double check the 
first part. `WMDE/Wikidata/Runbooks/Change dispatching/Alert` sounds like a 
good title to me!
  
  >> - https://wikitech.wikimedia.org/wiki/Performance/WebPageTest/Runbook/Alert
  >
  > This runbook happens to have been marked as historical in the meantime; the 
replacement seems to be 
https://wikitech.wikimedia.org/wiki/Performance/Guides/WebPageReplay_alert#WebPageReplay_alert_fired
 (another runbook, https://wikitech.wikimedia.org/wiki/WebPageReplay/Runbook, 
is for deploying new versions rather than reacting to an alert).
  
  Indeed, I did not find many runbooks about alerts specifically. Having the 
current version of that runbook is good, though it looks like not much has 
changed in the structure at first glance.
  
  >> So, I would suggest the following rough structure/sections for the Runbook:
  >
  > The second part could probably link to WMDE/Wikidata/Dispatching 
, unless you 
think there are parts we should summarize in the runbook? Rest of the structure 
sounds good to me.
  
  I agree. After reading through that documentation again, I don't think we can 
summarize it more in a way that is still useful. Linking to it is probably good 
enough. For now at least.
  
  > Moving back to ready for review in case @hoo wants to take a look too.
  
   (More input is appreciated, but it could also move forward with what there 
currently is.)

TASK DETAIL
  https://phabricator.wikimedia.org/T348443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Lucas_Werkmeister_WMDE, hoo, Aklapper, Michael, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T348443: Investigate incident runbook formats

2023-10-17 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE moved this task from In Peer Review to Ready for Peer 
Review on the Wikidata Dev Team (Wikidata.org Slice) board.
Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this 
task.
Lucas_Werkmeister_WMDE added subscribers: hoo, Lucas_Werkmeister_WMDE.
Lucas_Werkmeister_WMDE added a comment.


  > Looking at the naming of the pages in that category, I would suggest 
something like "Wikidata/Runbooks/Change dispatching" or 
"Wikidata/Runbooks/Change dispatching/Alert" as the page name.
  
  Given that we already have WMDE/Wikidata 
 I’d use that prefix (I’m 
not sure if you meant to imply that or not). Otherwise that sounds good to me. 
(I’d lean towards the latter option, with “alert” in the title.)
  
  > - https://wikitech.wikimedia.org/wiki/Performance/WebPageTest/Runbook/Alert
  
  This runbook happens to have been marked as historical in the meantime; the 
replacement seems to be 
https://wikitech.wikimedia.org/wiki/Performance/Guides/WebPageReplay_alert#WebPageReplay_alert_fired
 (another runbook, https://wikitech.wikimedia.org/wiki/WebPageReplay/Runbook, 
is for deploying new versions rather than reacting to an alert).
  
  > So, I would suggest the following rough structure/sections for the Runbook:
  
  The second part could probably link to WMDE/Wikidata/Dispatching 
, unless you 
think there are parts we should summarize in the runbook? Rest of the structure 
sounds good to me.
  
  Moving back to ready for review in case @hoo wants to take a look too.

TASK DETAIL
  https://phabricator.wikimedia.org/T348443

WORKBOARD
  https://phabricator.wikimedia.org/project/board/6751/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, hoo, Aklapper, Michael, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T348443: Investigate incident runbook formats

2023-10-17 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE claimed this task.
Lucas_Werkmeister_WMDE moved this task from Ready for Peer Review to In Peer 
Review on the Wikidata Dev Team (Wikidata.org Slice) board.

TASK DETAIL
  https://phabricator.wikimedia.org/T348443

WORKBOARD
  https://phabricator.wikimedia.org/project/board/6751/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Aklapper, Michael, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, 
Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T348443: Investigate incident runbook formats

2023-10-11 Thread Michael 
Michael moved this task from In Development to Ready for Peer Review on the 
Wikidata Dev Team (Wikidata.org Slice) board.
Michael removed Michael as the assignee of this task.
Michael added a comment.


  So, I would suggest the following rough structure/sections for the Runbook:
  
  1. create the wikitech page and clearly identify the potential alert that 
might be triggered, so that this page can also be found when searching for 
snippets from alert emails
  2. Overview of the Wikibase ChangeDispatching process: This should especially 
include: Where it is triggered? What tables are involved there? What jobs are 
triggered and where (repo wikis, clients), What are the "outputs" (updated 
values in articles, lines in watchlists, updated sitelinks on articles, ...)?
  3. describe how to differentiate between an issue where the root cause still 
seems to be active and still seems to be making things worse, and an issue 
where the cause seems to already have ceased, and the alert is due to delayed 
downstream effects.
  4. describe one way for how to maybe identify edits/editors causing issues 
based on the once case where we did so previously
  
  That is not exactly a great structure, but it seems it would make our current 
knowledge available, and it can be improved in the future. I would suggest that 
we created subtasks for each of the sections, with the first maybe being on the 
quick side of things.
  
  Thoughts?

TASK DETAIL
  https://phabricator.wikimedia.org/T348443

WORKBOARD
  https://phabricator.wikimedia.org/project/board/6751/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Aklapper, Michael, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, 
Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T348443: Investigate incident runbook formats

2023-10-09 Thread Michael 
Michael added a comment.


  This kind of document seems to be named "Runbook" at Wikimedia.
  
  There is a category for runbooks on wikitech: 
https://wikitech.wikimedia.org/wiki/Category:Runbooks - our document should 
probably be in that category as well.
  
  Looking at the naming of the pages in that category, I would suggest 
something like "Wikidata/Runbooks/Change dispatching" or 
"Wikidata/Runbooks/Change dispatching/Alert" as the page name.
  
  Overall, I'm not noticing an overarching structure in these runbooks.
  
  Examples for runbooks that seem relevant:
  
  - https://wikitech.wikimedia.org/wiki/Performance/WebPageTest/Runbook/Alert
  - 
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Cloud_VPS_alert_Puppet_failure_on
  - 
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Runbook_template
  
  Other somewhat related wikimedia documentation:
  
  - https://wikitech.wikimedia.org/wiki/Incident_response/Training
  - https://wikitech.wikimedia.org/wiki/Incident_response/Runbook

TASK DETAIL
  https://phabricator.wikimedia.org/T348443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Aklapper, Michael, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, 
Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T348443: Investigate incident runbook formats

2023-10-09 Thread Michael 
Michael renamed this task from "Investiage incident runbook formats" to 
"Investigate incident runbook formats".

TASK DETAIL
  https://phabricator.wikimedia.org/T348443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Aklapper, Michael, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, 
Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org