[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-13 Thread Maintenance_bot 
Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi, Maintenance_bot
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, Invadibot, Devnull, maantietaja, lmata, 
Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, 
Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331, 786, Suran38, Biggs657, Lalamarie69, Juan90264, Alter-paule, 
Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, 
Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-13 Thread jcrespo 
jcrespo added a comment.


Terminated Jobs:
 JobId  Level  FilesBytes   Status   FinishedName 

396417  Full 108,32011.70 G  OK   13-Dec-21 09:34 
graphite1004.eqiad.wmnet-Weekly-Mon-production-srv-carbon-whisper-daily
396418  Full 108,32011.70 G  OK   13-Dec-21 09:35 
graphite2003.codfw.wmnet-Weekly-Mon-production-srv-carbon-whisper-daily

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi, jcrespo
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-13 Thread jcrespo 
jcrespo added a comment.


Running Jobs:
Console connected using TLS at 13-Dec-21 09:20
 JobId  Type Level Files Bytes  Name  Status
==
396417  Back Full  4,568412.9 M 
graphite1004.eqiad.wmnet-Weekly-Mon-production-srv-carbon-whisper-daily is 
running
396418  Back Full  0 0  
graphite2003.codfw.wmnet-Weekly-Mon-production-srv-carbon-whisper-daily is 
running


TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi, jcrespo
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-13 Thread fgiunchedi 
fgiunchedi closed this task as "Resolved".
fgiunchedi claimed this task.
fgiunchedi added a comment.


  I'm tentatively resolving the task since all short term mitigations are 
completed, feel free to reopen if sth is amiss

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-13 Thread gerritbot 
gerritbot added a comment.


  Change 745838 **merged** by Filippo Giunchedi:
  
  [operations/puppet@production] graphite: backup 'daily' hierarchy, with 
weekly frequency, every Monday
  
  https://gerrit.wikimedia.org/r/745838

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread Manuel 
Manuel added a comment.


  In T294355#7563152 , 
@fgiunchedi wrote:
  
  > Happy to provide more guidance/info on T297494 
 as well though
  
  Thank you for the offer! We will come back to it!

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread fgiunchedi 
fgiunchedi added a comment.


  In T294355#7563057 , 
@Manuel wrote:
  
  > Thank you for the suggestion @fgiunchedi! Do we have an explanation 
somewhere of how to do this?
  
  Sure no problem! My understanding is that these metrics are published/pushed 
somewhat infrequently by background jobs, therefore a good starting point would 
be https://wikitech.wikimedia.org/wiki/Prometheus#Ephemeral_jobs_(Pushgateway) 
. Happy to provide more guidance/info on T297494 
 as well though

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread Manuel 
Manuel added a comment.


  Thank you for the suggestion @fgiunchedi! Do we have an explanation somewhere 
of how to do this?

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread fgiunchedi 
fgiunchedi added a comment.


  @Manuel @Lydia_Pintscher going forward I suggest also investing resources to 
switch to Prometheus as the supported metric system. Graphite is deprecated and 
in "life support" mode while all producers (essentially mediawiki and related) 
are being ported over, thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread Manuel 
Manuel removed a project: Wikidata-Campsite (Team A Hearth ).

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread Manuel 
Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread Manuel 
Manuel added a subscriber: Lydia_Pintscher.
Manuel added a comment.


  @Lydia_Pintscher: This is unfortunately really bad news: I have just 
discussed this issue with Lucas and Michael. The short version is that a 
maintenance attempt apparently failed badly and led to data loss (see incident 
report 
).
 There were no backups. As a result, we permanently lost data for all metrics 
that still lack historic data (11 October 2021 and earlier). Reconstructing the 
missing data from dumps is theoretically possible in some cases, but it will 
take a lot of developer resources. Our recommendation is to focus 
reconstruction attempts on only a very small number of extremely important 
metrics. If you have a shortlist of such candidates, the team would in the next 
step try to identify whether reconstruction is possible in theory and what 
would be needed to make it happen. Also, we are asking to get regular backups 
for the future (see T297145  and 
triggered by this also T297485 ).

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, 
Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, 
Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread jcrespo 
jcrespo added a comment.


  Let me give it a deeper look, while the patch by itself looks good as is, I 
want to check if a different (non-default) backup policy would be more 
advantageous in frequency and space. :-)

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread fgiunchedi 
fgiunchedi added a comment.


  In T294355#7559074 , 
@Lucas_Werkmeister_WMDE wrote:
  
  > In T294355#7531241 , 
@fgiunchedi wrote:
  >
  >> In T294355#7531236 , 
@Lucas_Werkmeister_WMDE wrote:
  >>
  >>> I’m not sure I understand the discussion correctly :) do you still need a 
list of paths to back up, or does it look like we can back up everything now?
  >>
  >> What's "everything" in this context? :) If you are talking about `daily` 
then yes it does look like it!
  >
  > I was thinking of everything, even non-daily stuff, but it looks like 
`daily` would actually be enough for us. Manuel created a list of important 
dashboards in T297145 ; the topics 
they use are:
  
  Ok! Please see the related review to start backing up `daily`. I've added 
@jcrespo too for signing off purposes

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread gerritbot 
gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, 
Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, 
Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, 
GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-10 Thread gerritbot 
gerritbot added a comment.


  Change 745838 had a related patch set uploaded (by Filippo Giunchedi; author: 
Filippo Giunchedi):
  
  [operations/puppet@production] graphite: backup 'daily' hierarchy
  
  https://gerrit.wikimedia.org/r/745838

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Invadibot, Devnull, maantietaja, lmata, Akuckartz, 
Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-09 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  In T294355#7531241 , 
@fgiunchedi wrote:
  
  > In T294355#7531236 , 
@Lucas_Werkmeister_WMDE wrote:
  >
  >> I’m not sure I understand the discussion correctly :) do you still need a 
list of paths to back up, or does it look like we can back up everything now?
  >
  > What's "everything" in this context? :) If you are talking about `daily` 
then yes it does look like it!
  
  I was thinking of everything, even non-daily stuff, but it looks like `daily` 
would actually be enough for us. Manuel created a list of important dashboards 
in T297145 ; the topics they use are:
  
  - https://grafana.wikimedia.org/d/00162/wikidata-site-stats?orgId=1
- daily.wikidata.site_stats.users
- daily.wikidata.site_stats.active_users.*
- daily.wikidata.site_stats.user_groups.*
- daily.wikidata.site_stats.total_edits
- daily.wikidata.site_stats.total_pages
- daily.wikidata.site_stats.good_articles
- daily.wikidata.showcaseItems
- daily.wikidata.site_stats.babel_users
- daily.wikidata.site_stats.language_usage.*
- daily.wikidata.site_stats.pagelinks_to_namespace.*
- daily.wikidata.site_stats.active_users_by_namespace.*
- daily.wikidata.site_stats.edits_by_namespace.*
- daily.wikidata.site_stats.active_items.1
  - 
https://grafana.wikimedia.org/d/00167/wikidata-datamodel?orgId=1=30m
- daily.wikidata.site_stats.pages_by_namespace.*
- daily.wikidata.site_stats.lexemes.wbl-forms
- daily.wikidata.site_stats.lexemes.wbl-senses
- daily.wikidata.site_stats.page_length.*
- daily.wikidata.datamodel.*
  - 
https://grafana.wikimedia.org/d/00175/wikidata-datamodel-statements?orgId=1=30m
- daily.wikidata.datamodel.item.statements.total
- daily.wikidata.datamodel.item.wb-identifiers.total
- daily.wikidata.datamodel.property.statements.total
- daily.wikidata.datamodel.property.wb-identifiers.total
- daily.wikidata.datamodel.instanceof.*
- daily.wikidata.datamodel.ranks.*
- daily.wikidata.datamodel.qualifiers
- daily.wikidata.datamodel.*.statements.*
  - 
https://grafana.wikimedia.org/d/UHV96YJGk/wikidata-datamodel-lexemes?orgId=1=1d
- daily.wikidata.datamodel.lexeme.languageItem.*
- daily.wikidata.datamodel.lexeme.lexicalCategoryItem.*
- daily.wikidata.datamodel.lexeme.languageCode.*
  - https://grafana.wikimedia.org/d/00560/wikidata-co-editors?orgId=1
- daily.monthly.wikidata.coeditors.*
- daily.monthly.wikidata.coeditors.*

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Invadibot, Devnull, maantietaja, lmata, Akuckartz, 
Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-12-09 Thread Manuel 
Manuel moved this task from Prioritized Wikidata Product Backlog (prioritised 
from top to bottom) to Team A Hearth  on the Wikidata-Campsite board.
Manuel edited projects, added Wikidata-Campsite (Team A Hearth ); removed 
Wikidata-Campsite.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

WORKBOARD
  https://phabricator.wikimedia.org/project/board/3402/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Invadibot, Devnull, maantietaja, lmata, Akuckartz, 
Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-26 Thread fgiunchedi 
fgiunchedi added a comment.


  In T294355#7531236 , 
@Lucas_Werkmeister_WMDE wrote:
  
  > I’m not sure I understand the discussion correctly :) do you still need a 
list of paths to back up, or does it look like we can back up everything now?
  
  What's "everything" in this context? :) If you are talking about `daily` then 
yes it does look like it!

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, maantietaja, lmata, 
Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, 
Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-26 Thread jcrespo 
jcrespo added a comment.


  I don't have the answer to that question, but whenever any of you have the 
servers and path(s), you can follow the instructions at 
https://wikitech.wikimedia.org/wiki/Bacula#Adding_a_new_client to send a 
preliminary backup proposal to Puppet, and I will assist you to merge it with 
the proper setup (e.g. schedule, day, etc.)  - I think it will be more useful 
to discuss the details over a patch :-).

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, maantietaja, lmata, 
Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, 
Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-26 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  I’m not sure I understand the discussion correctly :) do you still need a 
list of paths to back up, or does it look like we can back up everything now?

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, maantietaja, lmata, 
Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, 
Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-25 Thread fgiunchedi 
fgiunchedi added a comment.


  In T294355#7528880 , 
@jcrespo wrote:
  
  > One more question, to finally decide if setting up weekly full backups or 
daily but incremental- do all files mostly change completely, or only a subset 
of them? Incrementals are able to be done with file granularity only (it will 
backup fully files as long as its path or hash has changed), if value.wsp 
changes every minute, and there is only 1 per value, we will do "weekly only 
full", otherwise the daily incrementals may be preferred.
  
  My expectation is that most files we're backing up will change (otherwise it 
means the metric files are not being updated, which would make the backups less 
relevant) so definitely +1 for e.g. a weekly full backup
  
  > If we end up doing weekly fulls, 11GB * 12 weeks of retention = 130 GB, 
which we can handle with no issue.
  
  Thank you that's good to know!

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, 
lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, 
QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-25 Thread jcrespo 
jcrespo added a comment.


  One more question, to finally decide if setting up weekly full backups or 
daily but incremental- do all files mostly change completely, or only a subset 
of them? Incrementals are able to be done with file granularity only (it will 
backup fully files as long as its path or hash has changed), if value.wsp 
changes every minute, and there is only 1 per value, we will do "weekly only 
full", otherwise the daily incrementals may be preferred.
  
  If we end up doing weekly fulls, 11GB * 12 weeks of retention = 130 GB, which 
we can handle with no issue.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, 
lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, 
QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-25 Thread fgiunchedi 
fgiunchedi added a comment.


  In T294355#7527157 , 
@jcrespo wrote:
  
  > number of files are (within reason) a non-blocker for bacula, as files are 
packaged into volumes. It is true that each file is stored as a mysql record, 
but that should be able to scale until dozens of (US) billons, although it may 
be slow to recover when rebuilding metadata.
  >
  > Most limiting factor would be the overall size + backup frequency for 
capacity planning. We don't have a lot of temporal data backed up, so not sure 
if we could come up with a strategy that saves space (e.g. if data is 
immutable, we may want to avoid full backups every day). What is the 
file/directory structure? If data is below e.g. 100GB I would consider it 
"small" and not requiring optimization.
  >
  > The typical backup schedule is incrementals of a set of paths every day, 
differentials every fortnite, and fulls monthly- however it is highly 
customizable per job.
  
  Thank you that's helpful to know, my hunch is that we'd want every other week 
backups since this is mainly a safety measure. File structure is one file per 
metric for graphite, with the filesystem path mirroring the graphite path (e.g. 
`foo.bar.baz.value` will be `/foo/bar/baz/value.wsp` on the filesystem). All 
files are expected to be around ~100k (e.g. the `daily` top level directory I 
mentioned earlier is ~100k files and 11G in size).

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, 
lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, 
QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-24 Thread jcrespo 
jcrespo added projects: bacula, Data-Persistence-Backup, Data-Persistence.
jcrespo added a comment.


  number of files are (within reason) a non-blocker for bacula, as files are 
packaged into volumes. It is true that each file is stored as a mysql record, 
but that should be able to scale until dozens of (US) billons, although it may 
be slow to recover when rebuilding metadata.
  
  Most limiting factor would be the overall size + backup frequency for 
capacity planning. We don't have a lot of temporal data backed up, so not sure 
if we could come up with a strategy that saves space (e.g. if data is 
immutable, we may want to avoid full backups every day). What is the 
file/directory structure? If data is below e.g. 100GB I would consider it 
"small" and not requiring optimization.
  
  The typical backup schedule is incrementals of a set of paths every day, 
differentials every fortnite, and fulls monthly- however it is highly 
customizable per job.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, 
lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, 
QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-24 Thread fgiunchedi 
fgiunchedi added a subscriber: jcrespo.
fgiunchedi added a comment.


  In T294355#7527026 , 
@Lucas_Werkmeister_WMDE wrote:
  
  > Sounds like a good idea to me, I can’t judge how much would fit in Bacula. 
Do you need a list of important metrics (worth backing up)?
  
  Yes a list of "paths" in the metrics hierarchy would be greatly helpful.
  
  re: acceptable limits of files for bacula jobs I'm looping in @jcrespo for 
guidance/assistance

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, 
Lucas_Werkmeister_WMDE, Invadibot, maantietaja, lmata, Akuckartz, Nandana, 
Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-24 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  Sounds like a good idea to me, I can’t judge how much would fit in Bacula. Do 
you need a list of important metrics (worth backing up)?

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 
Invadibot, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, 
herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-23 Thread Manuel 
Manuel added a project: Wikidata-Campsite.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 
Invadibot, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, 
herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-23 Thread fgiunchedi 
fgiunchedi added a comment.


  I've sent the incident up for review, what do you think re: my proposal of 
adding parts of the hierarchy to bacula (if it is feasible in terms of number 
of files, e.g. `daily` is ~100k files now)

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 
Invadibot, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, 
herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-16 Thread lmata 
lmata added a project: SRE Observability (FY2021/2022-Q2).

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: lmata
Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 
Invadibot, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, 
herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-11-04 Thread Manuel 
Manuel added a project: Wikidata Analytics.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 
Invadibot, maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-10-29 Thread fgiunchedi 
fgiunchedi added a comment.


  Draft incident report: 
https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-10-29_graphite
  
  Please feel free to integrate/change as needed. I'll be OOO until the 18th 
and I'll pick this back up

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-10-28 Thread fgiunchedi 
fgiunchedi added a comment.


  Audit completed, what I did is count the number of null data points in the 
year leading to the graphite2003 reimage (i.e. the first reimage, where the 
backfill would have first failed) from 2020/10/14 to 2021/10/11 (first column). 
And the number of nulls after the first reimage (from 2021/10/12 to 2021/10/20) 
in the second column.
  
  The files for which backfill failed would have a high number of nulls in the 
first column but low in the second (i.e. datapoints are being appended now, but 
haven't for the last year). The full list for metrics with more than 10 nulls 
in last year but less than 10 in the last week is at 
https://people.wikimedia.org/~filippo/nulls-T294355 (12MB file). I believe 
these were all the metrics affected by the failed backfill and for which we 
lost historical data.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-10-28 Thread fgiunchedi 
fgiunchedi added a comment.


  Status update: I'm running a full audit on all ~4M metric files looking for 
similar cases. The backfill from yesterday completed in the mean time and some 
metrics were able to be backfilled successfully.
  
  I'll be following up with an incident report about this -- again my apologies 
for the unexpected data loss during migration and backfill.
  
  In terms of action items: we currently don't backup graphite metric files, 
mostly due to the sheer number of files and space they take. However if a 
subset of metric files in a directory hierarchy isn't too big (I don't have the 
exact number on hand for "big" but I'd say low tens of thousands, to be 
confirmed) then it should be doable to back up in bacula.

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-10-27 Thread fgiunchedi 
fgiunchedi added a comment.


  Status update: the backfill is still ongoing since I lowered the concurrency.
  
  The good news is that some metrics are already backfilled, e.g. api backend 
summary: 
https://grafana.wikimedia.org/d/2/api-backend-summary?viewPanel=31=1=161723520=163511000
  
  The bad news is that I suspect the first backfill on Oct 11th (i.e. when we 
reimaged and then backfilled graphite2003) suffered from the same undetected 
problem, therefore in that case we do have data loss unfortunately

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: fgiunchedi
Cc: Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-10-27 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-10-27T09:25:17Z]  
another run of backfill on graphite1004 - T294355 


TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

2021-10-27 Thread Aklapper 
Aklapper renamed this task from "Several Wikidata Grafana boards missing data 
before October 2022" to "Several Wikidata Grafana boards missing data before 
October 2021".

TASK DETAIL
  https://phabricator.wikimedia.org/T294355

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Aklapper
Cc: Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, 
maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org