[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi, Maintenance_bot Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore,

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

jcrespo added a comment. Terminated Jobs: JobId Level FilesBytes Status FinishedName 396417 Full 108,32011.70 G OK 13-Dec-21 09:34

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

jcrespo added a comment. Running Jobs: Console connected using TLS at 13-Dec-21 09:20 JobId Type Level Files Bytes Name Status == 396417 Back Full 4,568412.9 M

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi closed this task as "Resolved". fgiunchedi claimed this task. fgiunchedi added a comment. I'm tentatively resolving the task since all short term mitigations are completed, feel free to reopen if sth is amiss TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

gerritbot added a comment. Change 745838 **merged** by Filippo Giunchedi: [operations/puppet@production] graphite: backup 'daily' hierarchy, with weekly frequency, every Monday https://gerrit.wikimedia.org/r/745838 TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Manuel added a comment. In T294355#7563152 , @fgiunchedi wrote: > Happy to provide more guidance/info on T297494 as well though Thank you for the offer! We will come back to it! TASK

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. In T294355#7563057 , @Manuel wrote: > Thank you for the suggestion @fgiunchedi! Do we have an explanation somewhere of how to do this? Sure no problem! My understanding is that these metrics are

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Manuel added a comment. Thank you for the suggestion @fgiunchedi! Do we have an explanation somewhere of how to do this? TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc:

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. @Manuel @Lydia_Pintscher going forward I suggest also investing resources to switch to Prometheus as the supported metric system. Graphite is deprecated and in "life support" mode while all producers (essentially mediawiki and related) are being ported over,

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Manuel removed a project: Wikidata-Campsite (Team A Hearth ). TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore,

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Manuel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper,

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Manuel added a subscriber: Lydia_Pintscher. Manuel added a comment. @Lydia_Pintscher: This is unfortunately really bad news: I have just discussed this issue with Lucas and Michael. The short version is that a maintenance attempt apparently failed badly and led to data loss (see incident

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

jcrespo added a comment. Let me give it a deeper look, while the patch by itself looks good as is, I want to check if a different (non-default) backup policy would be more advantageous in frequency and space. :-) TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. In T294355#7559074 , @Lucas_Werkmeister_WMDE wrote: > In T294355#7531241 , @fgiunchedi wrote: > >> In T294355#7531236

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper,

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

gerritbot added a comment. Change 745838 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi): [operations/puppet@production] graphite: backup 'daily' hierarchy https://gerrit.wikimedia.org/r/745838 TASK DETAIL

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Lucas_Werkmeister_WMDE added a comment. In T294355#7531241 , @fgiunchedi wrote: > In T294355#7531236 , @Lucas_Werkmeister_WMDE wrote: > >> I’m not sure I understand the discussion

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Manuel moved this task from Prioritized Wikidata Product Backlog (prioritised from top to bottom) to Team A Hearth  on the Wikidata-Campsite board. Manuel edited projects, added Wikidata-Campsite (Team A Hearth ); removed Wikidata-Campsite. TASK DETAIL

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. In T294355#7531236 , @Lucas_Werkmeister_WMDE wrote: > I’m not sure I understand the discussion correctly :) do you still need a list of paths to back up, or does it look like we can back up everything now?

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

jcrespo added a comment. I don't have the answer to that question, but whenever any of you have the servers and path(s), you can follow the instructions at https://wikitech.wikimedia.org/wiki/Bacula#Adding_a_new_client to send a preliminary backup proposal to Puppet, and I will assist you

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Lucas_Werkmeister_WMDE added a comment. I’m not sure I understand the discussion correctly :) do you still need a list of paths to back up, or does it look like we can back up everything now? TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. In T294355#7528880 , @jcrespo wrote: > One more question, to finally decide if setting up weekly full backups or daily but incremental- do all files mostly change completely, or only a subset of them?

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

jcrespo added a comment. One more question, to finally decide if setting up weekly full backups or daily but incremental- do all files mostly change completely, or only a subset of them? Incrementals are able to be done with file granularity only (it will backup fully files as long as its

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. In T294355#7527157 , @jcrespo wrote: > number of files are (within reason) a non-blocker for bacula, as files are packaged into volumes. It is true that each file is stored as a mysql record, but that should

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

jcrespo added projects: bacula, Data-Persistence-Backup, Data-Persistence. jcrespo added a comment. number of files are (within reason) a non-blocker for bacula, as files are packaged into volumes. It is true that each file is stored as a mysql record, but that should be able to scale until

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a subscriber: jcrespo. fgiunchedi added a comment. In T294355#7527026 , @Lucas_Werkmeister_WMDE wrote: > Sounds like a good idea to me, I can’t judge how much would fit in Bacula. Do you need a list of important metrics

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Lucas_Werkmeister_WMDE added a comment. Sounds like a good idea to me, I can’t judge how much would fit in Bacula. Do you need a list of important metrics (worth backing up)? TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Manuel added a project: Wikidata-Campsite. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. I've sent the incident up for review, what do you think re: my proposal of adding parts of the hierarchy to bacula (if it is feasible in terms of number of files, e.g. `daily` is ~100k files now) TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

lmata added a project: SRE Observability (FY2021/2022-Q2). TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: lmata Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE,

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Manuel added a project: Wikidata Analytics. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Manuel Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. Draft incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-10-29_graphite Please feel free to integrate/change as needed. I'll be OOO until the 18th and I'll pick this back up TASK DETAIL

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. Audit completed, what I did is count the number of null data points in the year leading to the graphite2003 reimage (i.e. the first reimage, where the backfill would have first failed) from 2020/10/14 to 2021/10/11 (first column). And the number of nulls after

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. Status update: I'm running a full audit on all ~4M metric files looking for similar cases. The backfill from yesterday completed in the mean time and some metrics were able to be backfilled successfully. I'll be following up with an incident report about this

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

fgiunchedi added a comment. Status update: the backfill is still ongoing since I lowered the concurrency. The good news is that some metrics are already backfilled, e.g. api backend summary:

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-10-27T09:25:17Z] another run of backfill on graphite1004 - T294355 TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021

Aklapper renamed this task from "Several Wikidata Grafana boards missing data before October 2022" to "Several Wikidata Grafana boards missing data before October 2021". TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES