[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, Gehel Cc: Lydia_Pintscher, jcrespo, RKemper, bking, Oravrattas, Aklapper, Gehel, Astuthiodit_1,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. At long last..the data reload is complete! At this point, we believe the data discrepancies mentioned in the initial ticket message have been resolved. However, please let us know if this is not the case. TASK DETAIL https://phabricator.wikimedia.org/T323096

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, Maintenance_bot Cc: Lydia_Pintscher, jcrespo, RKemper, bking, Oravrattas, Aklapper,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 891899 **merged** by jenkins-bot: [operations/cookbooks@master] wdqs.data-transfer: completely replace defunct argument https://gerrit.wikimedia.org/r/891899 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 891899 had a related patch set uploaded (by Bking; author: Bking): [operations/cookbooks@master] wdqs.data-transfer: completely remove defunct argument https://gerrit.wikimedia.org/r/891899 TASK DETAIL https://phabricator.wikimedia.org/T323096

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, gerritbot Cc: Lydia_Pintscher, jcrespo, RKemper, bking, Oravrattas, Aklapper, Gehel,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, Maintenance_bot Cc: Lydia_Pintscher, jcrespo, RKemper, bking, Oravrattas, Aklapper,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 873791 **merged** by jenkins-bot: [operations/cookbooks@master] wdqs: make depool the default behavior https://gerrit.wikimedia.org/r/873791 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. The reload for `wdqs1010` completed successfully. We are using this host to seed other hosts using our data-transfer cookbook. We will update the ticket with progress. [ ] wdqs1003 [ ] wdqs1004 [ ] wdqs1005 [ ] wdqs1006 [ ] wdqs1007 [ ] wdqs1008 [X]

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel added a comment. For context, the journal corruptions we see are similar to T263110 (there was some investigation done at that time). Our only reasonable option is to try again until is succeeds. TASK DETAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. wdqs1009 corrupted itself over the weekend and we had to restart it. wdqs1010 at ~80%, wdqs1009 still in munging state. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. Update: we're at 80% on wdqs1009, 60% on wdqs1010 (unsure why 1009 is moving so much faster). TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Lydia_Pintscher,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. Another update: wdqs1009 corrupted itself over the weekend, so we have restarted the reload process again. wdqs1010 is still in progress. If we go by individual TTL files, it is at 56%, but I'm not sure if each TTL file takes the same amount of processing time. If so,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

MPhamWMF set the point value for this task to "13". TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, MPhamWMF Cc: Lydia_Pintscher, jcrespo, RKemper, bking, Oravrattas, Aklapper, Gehel,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. wdqs2010 locked up as well. At this point, we can rule out OOM. The most likely culprit is NFS, which is already scheduled to be replaced by rsync . More concerning is the failure of wdqs1009, which ran for 17 days and

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. In order to avoid the OOM situation mentioned above , I've created a large swapfile on `wdqs2010` at `/srv/.swapfile` . Making a note to myself to disable/remove this file once the reload is done. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel moved this task from Incoming to In Progress on the Discovery-Search (Current work) board. Gehel assigned this task to bking. TASK DETAIL https://phabricator.wikimedia.org/T323096 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel added a parent task: T322869: Fewer results from wdqs nodes running in codfw than eqiad. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: jcrespo, RKemper, bking, Oravrattas,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 882664 **merged** by Bking: [operations/puppet@production] wdqs: mount NFS to new hosts https://gerrit.wikimedia.org/r/882664 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 882664 had a related patch set uploaded (by Bking; author: Bking): [operations/puppet@production] wdqs: mount NFS to new hosts https://gerrit.wikimedia.org/r/882664 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. Update: wdqs2009 has failed to reload 3 times. Each time it seems to go into an OOM state where the server is pingable, but logging in (even from the management console) is impossible. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 876217 **merged** by Ryan Kemper: [operations/cookbooks@master] wdqs: use NFS for data reloads https://gerrit.wikimedia.org/r/876217 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 879636 **merged** by jenkins-bot: [wikidata/query/rdf@master] wdqs: log realpath of dump file https://gerrit.wikimedia.org/r/879636 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 879634 **abandoned** by Ryan Kemper: [wikidata/query/deploy@master] wdqs: log realpath of dump file Reason: wrong repo, see https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/879636 https://gerrit.wikimedia.org/r/879634 TASK DETAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 879636 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [wikidata/query/rdf@master] wdqs: log realpath of dump file https://gerrit.wikimedia.org/r/879636 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 879634 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [wikidata/query/deploy@master] wdqs: log realpath of dump file https://gerrit.wikimedia.org/r/879634 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

jcrespo added a comment. I was told by @Gehel that it was unrelated to this, but related to T301167 . Sorry for the confussion. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

jcrespo added a comment. Sorry if it is the wrong ticket, but several services of wdqs2010, wdqs2011 and wdqs2012 are alerting. The sevice is returnin 400 commands. My guess is this is due to this ongoing data reload (no issue). If that is the case, could the alerts "WDQS SPARQL" and other

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 876217 had a related patch set uploaded (by Bking; author: Bking): [operations/cookbooks@master] [WIP] wdqs-data-reload: use NFS for data reloads https://gerrit.wikimedia.org/r/876217 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. Data reload is currently running in tmux window "wdqs-reload" on cumin1001 under my user on wdqs2009 now. I'm using a custom version of the cookbook that loads via NFS. Not a permanent solution, but should allow us to complete our immediate task. TASK DETAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 873791 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/cookbooks@master] wdqs: make depool the default behavior https://gerrit.wikimedia.org/r/873791 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 869828 **merged** by Bking: [operations/puppet@production] query_service: add wdqs/wcqs hosts as rsync clients to clouddumps https://gerrit.wikimedia.org/r/869828 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: RKemper, bking, Oravrattas, Aklapper, Gehel, Adamm71, Jersione, Hellket777,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 869828 had a related patch set uploaded (by Bking; author: Bking): [operations/puppet@production] query_service: add wdqs/wcqs hosts as rsync clients to clouddumps https://gerrit.wikimedia.org/r/869828 TASK DETAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel added a parent task: T301167: Service implementation for wdqs20[09,10,11,12]. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: RKemper, bking, Oravrattas, Aklapper, Gehel,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel edited projects, added Discovery-Search (Current work); removed Wikidata-Query-Service. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: RKemper, bking, Oravrattas, Aklapper,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: RKemper, bking, Oravrattas, Aklapper, Gehel, Astuthiodit_1, AWesterinen,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 868440 **merged** by Bking: [operations/puppet@production] wdqs: allow mounting clouddumps share from wdqs2009 https://gerrit.wikimedia.org/r/868440 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: RKemper, bking, Oravrattas, Aklapper, Gehel, Adamm71, Jersione, Hellket777,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 868440 had a related patch set uploaded (by Bking; author: Bking): [operations/puppet@production] wdqs: allow mounting clouddumps share from wdqs2009 https://gerrit.wikimedia.org/r/868440 TASK DETAIL https://phabricator.wikimedia.org/T323096

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a comment. The following hosts now have the clouddumps NFS share mounted: - wcqs2001.codfw.wmnet - wdqs1009.eqiad.wmnet - wdqs2001.codfw.wmnet - wdqs2009.codfw.wmnet The ttl files we need are in

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: RKemper, bking, Oravrattas, Aklapper, Gehel, Astuthiodit_1, AWesterinen,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 867646 **merged** by Bking: [operations/puppet@production] Mount labstore to wcqs/wdqs instance for dumps reload https://gerrit.wikimedia.org/r/867646 TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: RKemper, bking, Oravrattas, Aklapper, Gehel, Adamm71, Jersione, Hellket777,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

gerritbot added a comment. Change 867646 had a related patch set uploaded (by Bking; author: Ebernhardson): [operations/puppet@production] Mount labstore to wcqs/wdqs instance for dumps reload https://gerrit.wikimedia.org/r/867646 TASK DETAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking closed subtask T305818: Perform a data transfer to wdqs2004 wdqs1004 to reclaim burnt allocators as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, bking,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a subtask: T305818: Perform a data transfer to wdqs2004 & wdqs1004 to reclaim burnt allocators. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, bking, Oravrattas,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

bking added a subscriber: RKemper. bking added a comment. @RKemper since we are doing data reloads on everything, I'm thinking maybe we should revisit this PR as it discusses a few ways to speed up the process TASK DETAIL

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel triaged this task as "High" priority. Gehel moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board. TASK DETAIL https://phabricator.wikimedia.org/T323096 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: bking, Oravrattas, Aklapper, Gehel, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: bking, Oravrattas, Aklapper, Gehel, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Aklapper, Gehel, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF,

[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload

Gehel created this task. Gehel added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION To address data discrepancy in WDQS, we need to reload data on all WDQS servers. In particular, this will address T322869