[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a comment. Change 833025 **abandoned** by DCausse: [wikidata/query/deploy@master] Remove deprecated wcqs-data-reload.sh script Reason: should be automatically removed during the next deploy https://gerrit.wikimedia.org/r/833025 TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a comment. Change 867646 **merged** by Bking: [operations/puppet@production] Mount labstore to wcqs/wdqs instance for dumps reload https://gerrit.wikimedia.org/r/867646 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a comment. Change 867646 had a related patch set uploaded (by Bking; author: Ebernhardson): [operations/puppet@production] Mount labstore to wcqs/wdqs instance for dumps reload https://gerrit.wikimedia.org/r/867646 TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, Gehel Cc: RKemper, dcaro, bking, EBernhardson, HenkvD, Aklapper, dcausse, Adamm71, Jersione,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking removed a subtask: T305818: Perform a data transfer to wdqs2004 & wdqs1004 to reclaim burnt allocators. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, dcaro, bking,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a subtask: T305818: Perform a data transfer to wdqs2004 & wdqs1004 to reclaim burnt allocators. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, dcaro, bking,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

RKemper reopened this task as "Open". RKemper added a comment. In T316236#8415706 , @RKemper wrote: > Moved to `Needs Reporting`. Usually we leave tickets open but move them to needs reporting and then gehel closes as resolved after he

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

RKemper moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. Moved to `Needs Reporting`. Usually we leave tickets open but move them to needs reporting and then gehel closes as resolved after he reviews them, but I'll

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking closed this task as "Resolved". bking moved this task from In Progress to Needs review on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T316236 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. Opened T323380 for the disk errors, closing this one for now. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc:

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. The reload is complete. However, we had to reboot `wcqs1003.eqiad.wmnet` several times before it would actually load the OS, and the BIOS displayed disk errors every time: Initializing Serial ATA devices... Port A: Device initialization error Port B:

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. wcqs1003 is the only host left that needs a reload: wcqs2003.codfw.wmnet | CHANGED | rc=0 >> /dev/mapper/vg0-srv 2.7T 470G 2.1T 19% /srv wcqs2002.codfw.wmnet | CHANGED | rc=0 >> /dev/mapper/vg0-srv 2.7T 470G 2.1T 19% /srv

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

ops-monitoring-bot added a comment. Icinga downtime and Alertmanager silence (ID=47a1387a-5ea2-4086-addb-7dc9cb66fddf) set by bking@cumin2002 for 7 days, 0:00:00 on 1 host(s) and their services with reason: data reload wcqs1003.eqiad.wmnet TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

RKemper removed a subtask: T322037: Data-transfer.py cookbook: stop prometheus-blazegraph-exporter service. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, RKemper Cc: dcaro, bking,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a subtask: T322037: Data-transfer.py cookbook: stop prometheus-blazegraph-exporter service. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: dcaro, bking, EBernhardson,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

ops-monitoring-bot added a comment. Icinga downtime and Alertmanager silence (ID=c3c6135a-4c18-4dff-b3d6-1cea09c7e58c) set by bking@cumin2002 for 5 days, 0:00:00 on 1 host(s) and their services with reason: data reload wcqs2003.codfw.wmnet TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

ops-monitoring-bot added a comment. Icinga downtime and Alertmanager silence (ID=b3b0c7c7-a37b-4722-8303-1e26dc50d1a3) set by bking@cumin2002 for 5 days, 0:00:00 on 1 host(s) and their services with reason: data reload wcqs2002.codfw.wmnet TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. Still not sure exactly why they're failling, wcqs2002 also has the new data. wcqs2003 and wcqs1003 still need it. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. As of this writing, the following hosts have the updated data: - wcqs2001 - wcqs1001 - wcqs1002 We can work on it further tomorrow. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

dcausse added a comment. @bking I did not spot any errors, the `not found, terminating` line is expected I guess. The reload cookbook seemed to have taken care of a bunch of the 3 first steps you mention, I think the remaining steps are just to propagate the journal using the

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. The data reload is complete, but there were errors . The gzipped output of the cookbook is available here . @dcausse , let us know if these errors are

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a comment. Change 835596 **merged** by Ryan Kemper: [operations/puppet@production] Mount labstore to wcqs/wdqs instance for dumps reload https://gerrit.wikimedia.org/r/835596 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. Upon further discussion with @EBernhardson , we'll hold off on the NFS changes for the time being and just load the dumps from HTTP. For the kafka timestamp, we prior sunday at midnight as a unix timestamp * 1000 ( 166587840) or an iso 8601 timestamp (

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. Apologies, as I lost track of this ticket. Repeating the list of steps from the initial message: 1. depool one wcqs2001 2. run the reload script 3. note the date of the dump and position the offset of the wcqs2001 consumer according to this date 4. start

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

EBernhardson updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, EBernhardson Cc: bking, EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

dcausse reassigned this task from EBernhardson to bking. dcausse added a comment. @bking what is the status of this ticket? TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, dcausse Cc:

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a comment. Change 835596 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] Mount labstore to wcqs/wdqs instance for dumps reload https://gerrit.wikimedia.org/r/835596 TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a comment. Change 832543 **merged** by Ryan Kemper: [operations/puppet@production] Mount labstore to wcqs/wdqs instance for dumps reload https://gerrit.wikimedia.org/r/832543 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

bking added a comment. This is currently running in tmux window `T316236` on `cumin2002` . TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, bking Cc: bking, EBernhardson, HenkvD,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

EBernhardson added a comment. To move this forward one of our SRE's will need to run the following and let it go for a couple days. After that the sre.wdqs.data-transfer cookbook will need to be used. cookbook sre.wdqs.data-reload wcqs2001.codw.wmnet \ --task-id T316236 \

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a comment. Change 833025 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [wikidata/query/deploy@master] Remove deprecated wcqs-data-reload.sh script https://gerrit.wikimedia.org/r/833025 TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

EBernhardson added a comment. The reload that was started on wcqs2001 didn't quite go right. We need to drop the reload scripts from the rdf deploy repo and only use the cookbooks going forward. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-09-15T22:01:45Z] END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wcqs2001.codfw.wmnet with reason: T316236 TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-09-15T22:01:20Z] START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wcqs2001.codfw.wmnet with reason: T316236 TASK DETAIL https://phabricator.wikimedia.org/T316236

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

EBernhardson moved this task from Ready for Development to In Progress on the Discovery-Search (Current work) board. EBernhardson claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T316236 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

EBernhardson added a comment. Also stopped wcqs-updater.service on wcqs2001, and disabled puppet so it wont be restarted TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc:

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-09-15T21:30:12Z] depool wcqs2001 for T316236 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

EBernhardson added a comment. Started download/munge on wcqs2001 using the internal dumps.wikimedia.org, we can't use dumps.wikimedia.your.org as it's dumps are two weeks out of date. The dumps are dated 20220911 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-09-15T19:26:29Z] pool'd wdqs2001, some blockers before reload can start T316236 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777, LisafBia6531,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

gerritbot added a comment. Change 832543 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [operations/puppet@production] Mount labstore to wcqs/wdqs instance for dumps reload https://gerrit.wikimedia.org/r/832543 TASK DETAIL

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

EBernhardson added a comment. Started looking into this, first problem is that dumps.wikimedia.your.org has changed their path layouts, a minor change to the data reload script will be necessary to pull from the correct paths and not 404. As long as we are revisiting this script though, it

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-09-15T18:15:50Z] depool wcqs2001 for T316236 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Gehel set the point value for this task to "5". TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: HenkvD, Aklapper, dcausse, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

dcausse updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: HenkvD, Aklapper, dcausse, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board. Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T316236 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Aklapper, dcausse, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF,

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

dcausse added a parent task: T314703: Structured data for deleted files on Commons still visible in SPARQL engine after deletion. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc:

[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps

dcausse created this task. dcausse added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Folluwup on T314703 where the consumer side of the updater was misconfigured and caused all deletes to