[Wikidata-bugs] [Maniphest] [Created] T252068: WQDS Data Reload

2020-05-06 Thread RKemper
RKemper created this task. RKemper added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION We have already done the reload on `wdqs1010`, so now we need to xfer to the various other

[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

2020-05-06 Thread RKemper
RKemper added a comment. `wdqs1010` is one of our test servers Therefore can screw up `wdqs1010` as much as we want, but not the others --- we need to transfer that data to each server which implies we need to open a port for data transfer we have 2 primary DC: eqiad

[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

2020-05-06 Thread RKemper
RKemper added a comment. should also circle back and fix https://github.com/wikimedia/puppet/blob/production/manifests/site.pp#L2207-L2242,` wdqs200[78]` are declared twice TASK DETAIL https://phabricator.wikimedia.org/T252068 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Unblock] T230588: Wikidata Query Service is swapping items and properties

2020-05-19 Thread RKemper
RKemper closed subtask T252068: WQDS Data Reload as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T230588 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, RKemper Cc: Ghuron, Nikki, Toni_001, Dipsacus_fullonum, Larske, Mathew.onipe

[Wikidata-bugs] [Maniphest] [Unblock] T245135: Too many values for wikibase:geoLatitude and wikibase:geoLongitude in WDQS

2020-05-19 Thread RKemper
RKemper closed subtask T252068: WQDS Data Reload as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T245135 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Lucas_Werkmeister_WMDE, Aklapper, Dipsacus_fullonum, CBogen, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Unblock] T240831: Sometimes Q77104211 shows up in sparql query as having a wrong value for P2397, however item is deleted since December 3rd 2019

2020-05-19 Thread RKemper
RKemper closed subtask T252068: WQDS Data Reload as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T240831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Aklapper, Mbch331, CBogen, darthmon_wmde, Nandana, Lahi, Gq86

[Wikidata-bugs] [Maniphest] [Closed] T252068: WQDS Data Reload

2020-05-19 Thread RKemper
RKemper closed this task as "Resolved". RKemper added a comment. Still need to circle back to dependent tickets and verify that those problems are solved TASK DETAIL https://phabricator.wikimedia.org/T252068 EMAIL PREFERENCES https://phabricator.wikimedia.org/sett

[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

2020-05-19 Thread RKemper
RKemper added a comment. Data transfer is done across all instances as of last friday, along with a wdqs-categories reload that we tacked on. Circling back to mark this ticket as done. TASK DETAIL https://phabricator.wikimedia.org/T252068 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T261204: Wikidata lexeme ttl dumps should be in a "predictable" folder

2020-09-12 Thread RKemper
RKemper added a comment. dumpsgen 58563 0.0 0.0 4276 700 ?Ss 20:50 0:00 /bin/sh -c python3 /srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py --configfile /etc/dumps/confs/addschanges.conf --dumptype incrdumps --quiet dumpsgen 58565 0.0 0.0 57976

[Wikidata-bugs] [Maniphest] T261204: Wikidata lexeme ttl dumps should be in a "predictable" folder

2020-09-12 Thread RKemper
RKemper moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. We'll want to monitor across the next week to verify all looks good. TASK DETAIL https://phabricator.wikimedia.org/T261204 WORKBOARD https

[Wikidata-bugs] [Maniphest] T262009: Add an entry in the WDQS Runbook on killer queries

2020-10-14 Thread RKemper
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T262009 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, Gehel, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86

[Wikidata-bugs] [Maniphest] T265868: wdqs2002 out of disk space

2020-10-19 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. This should be resolved now; the transfer is complete and the node matches others in the fleet. ryankemper@wdqs2002:~$ ls -lh /srv/wdqs/wikidata.jnl

[Wikidata-bugs] [Maniphest] T258835: Update Commons query service favicon

2020-09-24 Thread RKemper
RKemper moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. Applied this today TASK DETAIL https://phabricator.wikimedia.org/T258835 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

2020-07-27 Thread RKemper
RKemper added a comment. We discussed this during this week's SRE meeting and resolved to enable full root access for wdqs admins, rather than granularly expanding access one file at a time. This also lines up better with how we currently manage Elasticsearch, where admins have root access

[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

2020-07-23 Thread RKemper
RKemper added a project: Sustainability (Incident Followup). TASK DETAIL https://phabricator.wikimedia.org/T258739 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, CDanis, dcausse, Aklapper, Dzahn, lmata, Alter-paule, Beast1978

[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

2020-07-23 Thread RKemper
RKemper renamed this task from "wdqs admins should have access to nginx logs on wdqs machines" to "wdqs admins should have access to nginx logs, jstack on wdqs machines". TASK DETAIL https://phabricator.wikimedia.org/T258739 EMAIL PREFERENCES https://phabricator.wi

[Wikidata-bugs] [Maniphest] T271412: Investigate an alert on statements volume in Blazegraph instances

2021-01-11 Thread RKemper
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T271412 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, Zbyszko, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86

[Wikidata-bugs] [Maniphest] T269302: Many files on Commons cannot be found in WCQS

2020-12-07 Thread RKemper
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T269302 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Lydia_Pintscher, Vojtech.dostal, Dipsacus_fullonum, Aklapper, Muchiri124, CBogen, Nintendofan885

[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

2020-12-03 Thread RKemper
RKemper added a comment. `sudo -i wmf-auto-reimage-host --conftool -p T269204 wdqs2004.codfw.wmnet` is an example of how to reimage hosts (run from cumin) TASK DETAIL https://phabricator.wikimedia.org/T269204 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

2020-12-03 Thread RKemper
RKemper claimed this task. RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T269204 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, RKemper, dcausse, Aklapper, lmata, CBogen, Akuckartz, Nandana

[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

2020-12-03 Thread RKemper
RKemper added a comment. Note: https://phabricator.wikimedia.org/T269331 was created due to an exception encountered every time we run the data reload cookbook. The exception only occurs at the very end, so it might not indicate an actual problem TASK DETAIL https

[Wikidata-bugs] [Maniphest] T269331: wdqs.data-reload cookbook fails when switching

2020-12-03 Thread RKemper
RKemper added a comment. Here's something interesting: On a server where I've recently run the `data-reload` for `categories` (`wdqs1006` in this case), the categories journal looks like so: `-rw-rw-r-- 1 blazegraph blazegraph 21G Dec 3 07:19 categories.jnl` Yet on a wdqs

[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

2020-12-08 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. All categories reloads/transfers are complete TASK DETAIL https://phabricator.wikimedia.org/T259588 WORKBOARD https://phabricator.wikimedia.org/project

[Wikidata-bugs] [Maniphest] T269693: mediawiki_job_wikidata-updateQueryServiceLag failing

2020-12-09 Thread RKemper
RKemper added a comment. There's some context in the description of https://phabricator.wikimedia.org/T269204 that mentions that the counter metric `blazegraph_lastupdated` is now `blazegraph_lastupdated_total`, so if the`mediawiki_job_wikidata-updateQueryServiceLag` job has to do

[Wikidata-bugs] [Maniphest] T269693: mediawiki_job_wikidata-updateQueryServiceLag failing

2020-12-09 Thread RKemper
RKemper added a comment. Job lives here: https://github.com/wikimedia/mediawiki-extensions-Wikidata.org/blob/60c5f96ebf424b792077bb7c6b533a68702e7aea/maintenance/updateQueryServiceLag.php#L70 I have a patch open here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/646888

[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

2020-11-30 Thread RKemper
RKemper moved this task from Ready for Development to In Progress on the Discovery-Search (Current work) board. RKemper added a comment. Will start on this today TASK DETAIL https://phabricator.wikimedia.org/T259588 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL

[Wikidata-bugs] [Maniphest] T244753: Migrate WDQS to Debian Buster

2020-12-17 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. This is all done. No issues currently. TASK DETAIL https://phabricator.wikimedia.org/T244753 WORKBOARD https://phabricator.wikimedia.org/project/board/1227

[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on wdqs1011

2020-12-10 Thread RKemper
RKemper created this task. RKemper added projects: Wikidata-Query-Service, Discovery-Search (Current work). Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION ryankemper@wdqs1011:~$ sudo systemctl status prometheus

[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on select wdqs instances

2020-12-10 Thread RKemper
RKemper renamed this task from "wdqs-categories prometheus exporter failing on wdqs1011" to "wdqs-categories prometheus exporter failing on select wdqs instances". RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T269872 EMAIL

[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

2020-12-14 Thread RKemper
RKemper added a comment. Restarting `prometheus-blazegraph-exporter-wdqs-blazegraph.service` after switching from Counter to Gauge now shows the correct `blazegraph_lastupdated` metric when running `curl localhost:9193` (9193 is the port for wdqs-blazegraph). Still waiting to see

[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on select wdqs instances

2020-12-14 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. The above two patches (really one patch with a followup patch to fix a typo) seem to have fixed the problem. I'll want to circle back to verify we're getting

[Wikidata-bugs] [Maniphest] T255399: Prepare wdqs1009 to run the streaming updater

2020-11-16 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. We need to do another reload but not under this ticket TASK DETAIL https://phabricator.wikimedia.org/T255399 WORKBOARD https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] T271851: Clean up gui from the wdqs deploy repo and puppet

2021-01-21 Thread RKemper
RKemper added a comment. In T271851#6765207 <https://phabricator.wikimedia.org/T271851#6765207>, @Ladsgroup wrote: > @RKemper I'm so sorry it was buggy, the cleanest way forward here is to deploy wcqs using microsites and we can easily drop everything from puppet then. Does

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

2021-01-19 Thread RKemper
RKemper moved this task from In Progress to To Be Deployed on the Discovery-Search (Current work) board. RKemper added a comment. Deploying https://gerrit.wikimedia.org/r/656833 now TASK DETAIL https://phabricator.wikimedia.org/T267175 WORKBOARD https://phabricator.wikimedia.org/project

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

2021-01-19 Thread RKemper
RKemper added a comment. (Puppet run successful on wdqs nodes following deploy) TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Skim, Strepon, Multichill, Zbyszko, RKemper

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

2021-01-20 Thread RKemper
RKemper added a comment. Post deploy check: `wdqs-updater.service` is now running without the `--import-async` flag as expected: `ExecStart=/bin/bash /srv/deployment/wdqs/wdqs/runUpdate.sh -n wdq -- --kafka kafka-main1001.eqiad.wmnet:9092,kafka-main1002.eqiad.wmnet:9092,kafka

[Wikidata-bugs] [Maniphest] T270476: Linked Data Fragments endpoint returns IllegalStateException

2021-01-12 Thread RKemper
RKemper added a comment. I responded to Maciej (the one who reported the problem originally) on the mailing list. Looks like we accidentally gave them the wrong ticket number earlier: https://lists.wikimedia.org/pipermail/wikidata/2021-January/014451.html Looks like this ticket hasn't

[Wikidata-bugs] [Maniphest] T270236: wdqs1005: disk space critical on /srv/

2020-12-16 Thread RKemper
RKemper added a project: Wikidata-Query-Service. Restricted Application added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T270236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, MPhamWMF, CBogen

[Wikidata-bugs] [Maniphest] T270236: wdqs1005: disk space critical on /srv/

2020-12-16 Thread RKemper
RKemper triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T270236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, L

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-01 Thread RKemper
RKemper claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE

[Wikidata-bugs] [Maniphest] T273097: Create Flink Base Image

2021-02-03 Thread RKemper
RKemper added a comment. @akosiaris Is your concern with the idea of using a`flink` base image solution mainly just centered around the inefficiency/inconvenience of needing SRE to merge any flink version upgrades? Since we have an embedded SRE on search (me) and to a lesser extent

[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-03 Thread RKemper
RKemper added a comment. Notified WikiData mailing list and also posted here: https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service#Updates TASK DETAIL https://phabricator.wikimedia.org/T273636 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-03 Thread RKemper
RKemper added a comment. We'll want to reload these this Friday, because the latest dumps should be available thursday evening. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc

[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-03 Thread RKemper
RKemper added a comment. `wcqs-beta-01.eqiad.wmflabs` is running low on disk space due to its blazegraph journal dataset size. In order to free up space we will need to take the service down, delete the journal and re-import from the latest dump. Service interruption will begin at Feb 4 18

[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T273636 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86

[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. WCQS is back in service; updating the notification channels right now and will comment back here after TASK DETAIL https://phabricator.wikimedia.org/T273636

[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper added a comment. Updated: [X] https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service#Updates [X] https://commons.wikimedia.org/wiki/Commons:Village_pump#Unscheduled_maintenance%3A_Wikimedia_Commons_Query_Service [X] WikiData ML mailing list (note: A community member

[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink

2021-02-08 Thread RKemper
RKemper added a comment. (See https://phabricator.wikimedia.org/T273097#6805355 for why this ticket has been closed) TASK DETAIL https://phabricator.wikimedia.org/T266495 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-08 Thread RKemper
RKemper added a comment. Still waiting for the latest dumps to be downloaded (few more hours), then need to reboot WDQS hosts as part of https://phabricator.wikimedia.org/T274213, then can do the actual data-reload TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-02-01 Thread RKemper
RKemper triaged this task as "Medium" priority. RKemper claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamW

[Wikidata-bugs] [Maniphest] T273097: Create Flink Base Image

2021-02-01 Thread RKemper
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T273097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: dcausse, Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-05 Thread RKemper
RKemper added a comment. `sudo cookbook sre.wdqs.data-reload wdqs1009.eqiad.wmnet --reuse-downloaded-dump --reload-data wikidata --skolemize --reason 'T267927: Reload wikidata jnl from fresh dumps' --task-id T267927 ` is failing with: - OUTPUT of 'test -f /srv/wdq...test

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-02-04 Thread RKemper
RKemper added a comment. TODO from IRC meeting with bblack/gehel: create a DNS entry (CNAME to dyna.wm.o), another set of entries in backend.yaml map, create another minisite (with the appropriate configuration) TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-25 Thread RKemper
RKemper added a comment. Downtimed `wdqs2008` until `2021-03-04 21:56:59` TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Zbyszko, Gehel, Aklapper, dcausse, MPhamWMF, CBogen

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-25 Thread RKemper
RKemper added a comment. Note: Puppet is still disabled on `wdqs2008` while the reload runs. It occurred to me that I'm not sure if puppet actually **needs** to be disabled during data reloads or if that's just a precaution we've historically taken - any insight here @Gehel? TASK DETAIL

[Wikidata-bugs] [Maniphest] T274751: Upgrade firmware on wdqs1009

2021-02-25 Thread RKemper
RKemper added a comment. @Cmjohnson The data reload is complete on `wdqs1009`, so the host can now have its firmware upgraded and be rebooted at its convenience. Note this is an internal wdqs test host, so there is no public-facing service for us to worry about. Feel free to proceed

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-04 Thread RKemper
RKemper added a comment. Here's how the ats mapping looks afterdeploy of the `backend.yaml` changes: ryankemper@cp1075:~$ sudo cat /etc/trafficserver/remap.config | grep query map http://query-preview.wikidata.org/bigdata https://wdqs1009.eqiad.wmnet/bigdata map http://query

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-03 Thread RKemper
RKemper added a comment. Posting logs of our IRC convo from ~1 month ago for context when I tag people for review: [2021-02-04 11:31:36] bblack: around ? [2021-02-04 11:31:58] I'm here with ryankemper to talk about T266470, unless you want to jump in a meet [2021-02-04 11:31

[Wikidata-bugs] [Maniphest] T252504: Automate the smoke test of the canary deployment of WDQS

2021-03-08 Thread RKemper
RKemper renamed this task from "Smoke test the canary deployment of WDQS" to "Automate the smoke test of the canary deployment of WDQS". RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T252504 EMAIL PREFERENCES https://phabricator.wi

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment. I missed a step yesterday: I'd updated `/srv/private` as well as the public `labs/private` repo but missed the step for updating `operations/puppet` with the new pubkey: https://gerrit.wikimedia.org/r/c/operations/puppet/+/671267 TASK DETAIL https

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment. The issues with envoy were resolved by running `sudo /usr/local/sbin/build-envoy-config -c /etc/envoy` to properly build `/etc/envoy/envoy.yaml`. That should have been done by puppet already, triggered upon a `sudo systemctl restart envoyproxy.service

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment. **DNS change logs** ryankemper@authdns1001:~$ sudo authdns-update Updating authdns1001.wikimedia.org (self)... Pulling the current revision from https://gerrit.wikimedia.org/r/operations/dns.git Reviewing 85d9b49dc2ff0f8e3657f6f2cd91ce3df79bd1cf

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment. **DNS change logs** ryankemper@authdns1001:~$ sudo authdns-update Updating authdns1001.wikimedia.org (self)... Pulling the current revision from https://gerrit.wikimedia.org/r/operations/dns.git Reviewing 85d9b49dc2ff0f8e3657f6f2cd91ce3df79bd1cf

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-10 Thread RKemper
RKemper added a comment. Finished rolling back to the previous iteration of `wdqs.discovery.wmnet` cert since we're now going to create a net-new cert `wdqs1009.eqiad.wmnet` for `wdqs-test` TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-10 Thread RKemper
RKemper added a comment. Certificate(wdqs.discovery.wmnet, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]): /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.private.pem: PRESENT (mtime: 2019-10-21T08:11:52.741452

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment. Current status for when I pick this back up: - `query-preview.wikidata.org` (with `wdqs1009.eqiad.wmnet` as an `alt_name`) is generated - reverted https://gerrit.wikimedia.org/r/c/operations/puppet/+/670339/ for now, envoy wasn't happy: Error: /Stage

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment. Hit a big blocker with the current proposed approach of using `wdqs1009.eqiad.wmnet` as the cert name: ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wdqs1009.*' --generate --base-path /srv/private/modules/secret/secrets/certificates /srv/private

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment. **Option 2 ** fails to even generate the cert. All the cergen documentation is written for a certificate like `query-preview.discovery.wmnet` and not `wdqs1009.eqiad.wmnet` or `query-preview.wikidata.org`. So I do think this just isn't what cergen is built to do

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment. Ah, so poking around the `certificate.manifests.d` repo I see certs that don't necessarily follow the `discovery.wmnet` pattern. To me that implies **Option 2** should be working, so I might be missing something. Here's an example that doesn't use discovery

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-09 Thread RKemper
RKemper added a comment. Certificate(wdqs.discovery.wmnet, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]): /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.private.pem: PRESENT (mtime: 2019-10-21T08:11:52.741452

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-03-24 Thread RKemper
RKemper added a comment. `wdqs1009`, `wdqs1010`, and `wdqs2008` are done, so we need to `data-transfer` to the remaining instances. It looks like `wdqs1010` is warning (not critical) for disk space on `/srv`. Glancing at the box it's still got 44G left but is at 96% utilization