[Wikidata-bugs] [Maniphest] T271851: Clean up gui from the wdqs deploy repo and puppet

RKemper added a comment. In T271851#6765207 <https://phabricator.wikimedia.org/T271851#6765207>, @Ladsgroup wrote: > @RKemper I'm so sorry it was buggy, the cleanest way forward here is to deploy wcqs using microsites and we can easily drop everything from puppet then. Does

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

RKemper added a comment. Post deploy check: `wdqs-updater.service` is now running without the `--import-async` flag as expected: `ExecStart=/bin/bash /srv/deployment/wdqs/wdqs/runUpdate.sh -n wdq -- --kafka kafka-main1001.eqiad.wmnet:9092,kafka-main1002.eqiad.wmnet:9092,kafka

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

RKemper added a comment. (Puppet run successful on wdqs nodes following deploy) TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Skim, Strepon, Multichill, Zbyszko, RKemper

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

RKemper moved this task from In Progress to To Be Deployed on the Discovery-Search (Current work) board. RKemper added a comment. Deploying https://gerrit.wikimedia.org/r/656833 now TASK DETAIL https://phabricator.wikimedia.org/T267175 WORKBOARD https://phabricator.wikimedia.org/project

[Wikidata-bugs] [Maniphest] T270476: Linked Data Fragments endpoint returns IllegalStateException

RKemper added a comment. I responded to Maciej (the one who reported the problem originally) on the mailing list. Looks like we accidentally gave them the wrong ticket number earlier: https://lists.wikimedia.org/pipermail/wikidata/2021-January/014451.html Looks like this ticket hasn't

[Wikidata-bugs] [Maniphest] T271412: Investigate an alert on statements volume in Blazegraph instances

RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T271412 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, Zbyszko, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86

[Wikidata-bugs] [Maniphest] T244753: Migrate WDQS to Debian Buster

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. This is all done. No issues currently. TASK DETAIL https://phabricator.wikimedia.org/T244753 WORKBOARD https://phabricator.wikimedia.org/project/board/1227

[Wikidata-bugs] [Maniphest] T270236: wdqs1005: disk space critical on /srv/

RKemper triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T270236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, L

[Wikidata-bugs] [Maniphest] T270236: wdqs1005: disk space critical on /srv/

RKemper added a project: Wikidata-Query-Service. Restricted Application added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T270236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, MPhamWMF, CBogen

[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on select wdqs instances

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. The above two patches (really one patch with a followup patch to fix a typo) seem to have fixed the problem. I'll want to circle back to verify we're getting

[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

RKemper added a comment. Restarting `prometheus-blazegraph-exporter-wdqs-blazegraph.service` after switching from Counter to Gauge now shows the correct `blazegraph_lastupdated` metric when running `curl localhost:9193` (9193 is the port for wdqs-blazegraph). Still waiting to see

[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on select wdqs instances

RKemper renamed this task from "wdqs-categories prometheus exporter failing on wdqs1011" to "wdqs-categories prometheus exporter failing on select wdqs instances". RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T269872 EMAIL

[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on wdqs1011

RKemper created this task. RKemper added projects: Wikidata-Query-Service, Discovery-Search (Current work). Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION ryankemper@wdqs1011:~$ sudo systemctl status prometheus

[Wikidata-bugs] [Maniphest] T269693: mediawiki_job_wikidata-updateQueryServiceLag failing

RKemper added a comment. Job lives here: https://github.com/wikimedia/mediawiki-extensions-Wikidata.org/blob/60c5f96ebf424b792077bb7c6b533a68702e7aea/maintenance/updateQueryServiceLag.php#L70 I have a patch open here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/646888

[Wikidata-bugs] [Maniphest] T269693: mediawiki_job_wikidata-updateQueryServiceLag failing

RKemper added a comment. There's some context in the description of https://phabricator.wikimedia.org/T269204 that mentions that the counter metric `blazegraph_lastupdated` is now `blazegraph_lastupdated_total`, so if the`mediawiki_job_wikidata-updateQueryServiceLag` job has to do

[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. All categories reloads/transfers are complete TASK DETAIL https://phabricator.wikimedia.org/T259588 WORKBOARD https://phabricator.wikimedia.org/project

[Wikidata-bugs] [Maniphest] T269302: Many files on Commons cannot be found in WCQS

RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T269302 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Lydia_Pintscher, Vojtech.dostal, Dipsacus_fullonum, Aklapper, Muchiri124, CBogen, Nintendofan885

[Wikidata-bugs] [Maniphest] T269331: wdqs.data-reload cookbook fails when switching

RKemper added a comment. Here's something interesting: On a server where I've recently run the `data-reload` for `categories` (`wdqs1006` in this case), the categories journal looks like so: `-rw-rw-r-- 1 blazegraph blazegraph 21G Dec 3 07:19 categories.jnl` Yet on a wdqs

[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

RKemper added a comment. Note: https://phabricator.wikimedia.org/T269331 was created due to an exception encountered every time we run the data reload cookbook. The exception only occurs at the very end, so it might not indicate an actual problem TASK DETAIL https

[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

RKemper claimed this task. RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T269204 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, RKemper, dcausse, Aklapper, lmata, CBogen, Akuckartz, Nandana

[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

RKemper added a comment. `sudo -i wmf-auto-reimage-host --conftool -p T269204 wdqs2004.codfw.wmnet` is an example of how to reimage hosts (run from cumin) TASK DETAIL https://phabricator.wikimedia.org/T269204 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

RKemper moved this task from Ready for Development to In Progress on the Discovery-Search (Current work) board. RKemper added a comment. Will start on this today TASK DETAIL https://phabricator.wikimedia.org/T259588 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL

[Wikidata-bugs] [Maniphest] T255399: Prepare wdqs1009 to run the streaming updater

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. We need to do another reload but not under this ticket TASK DETAIL https://phabricator.wikimedia.org/T255399 WORKBOARD https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] T265868: wdqs2002 out of disk space

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. This should be resolved now; the transfer is complete and the node matches others in the fleet. ryankemper@wdqs2002:~$ ls -lh /srv/wdqs/wikidata.jnl

[Wikidata-bugs] [Maniphest] T262009: Add an entry in the WDQS Runbook on killer queries

RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T262009 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, Gehel, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86

[Wikidata-bugs] [Maniphest] T258835: Update Commons query service favicon

RKemper moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. Applied this today TASK DETAIL https://phabricator.wikimedia.org/T258835 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T261204: Wikidata lexeme ttl dumps should be in a "predictable" folder

RKemper moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. We'll want to monitor across the next week to verify all looks good. TASK DETAIL https://phabricator.wikimedia.org/T261204 WORKBOARD https

[Wikidata-bugs] [Maniphest] T261204: Wikidata lexeme ttl dumps should be in a "predictable" folder

RKemper added a comment. dumpsgen 58563 0.0 0.0 4276 700 ?Ss 20:50 0:00 /bin/sh -c python3 /srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py --configfile /etc/dumps/confs/addschanges.conf --dumptype incrdumps --quiet dumpsgen 58565 0.0 0.0 57976

[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

RKemper added a comment. We discussed this during this week's SRE meeting and resolved to enable full root access for wdqs admins, rather than granularly expanding access one file at a time. This also lines up better with how we currently manage Elasticsearch, where admins have root access

[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

RKemper added a project: Sustainability (Incident Followup). TASK DETAIL https://phabricator.wikimedia.org/T258739 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, CDanis, dcausse, Aklapper, Dzahn, lmata, Alter-paule, Beast1978

[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

RKemper renamed this task from "wdqs admins should have access to nginx logs on wdqs machines" to "wdqs admins should have access to nginx logs, jstack on wdqs machines". TASK DETAIL https://phabricator.wikimedia.org/T258739 EMAIL PREFERENCES https://phabricator.wi

[Wikidata-bugs] [Maniphest] [Unblock] T230588: Wikidata Query Service is swapping items and properties

RKemper closed subtask T252068: WQDS Data Reload as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T230588 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, RKemper Cc: Ghuron, Nikki, Toni_001, Dipsacus_fullonum, Larske, Mathew.onipe

[Wikidata-bugs] [Maniphest] [Unblock] T240831: Sometimes Q77104211 shows up in sparql query as having a wrong value for P2397, however item is deleted since December 3rd 2019

RKemper closed subtask T252068: WQDS Data Reload as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T240831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Aklapper, Mbch331, CBogen, darthmon_wmde, Nandana, Lahi, Gq86

[Wikidata-bugs] [Maniphest] [Closed] T252068: WQDS Data Reload

RKemper closed this task as "Resolved". RKemper added a comment. Still need to circle back to dependent tickets and verify that those problems are solved TASK DETAIL https://phabricator.wikimedia.org/T252068 EMAIL PREFERENCES https://phabricator.wikimedia.org/sett

[Wikidata-bugs] [Maniphest] [Unblock] T245135: Too many values for wikibase:geoLatitude and wikibase:geoLongitude in WDQS

RKemper closed subtask T252068: WQDS Data Reload as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T245135 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Lucas_Werkmeister_WMDE, Aklapper, Dipsacus_fullonum, CBogen, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

RKemper added a comment. Data transfer is done across all instances as of last friday, along with a wdqs-categories reload that we tacked on. Circling back to mark this ticket as done. TASK DETAIL https://phabricator.wikimedia.org/T252068 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

RKemper added a comment. should also circle back and fix https://github.com/wikimedia/puppet/blob/production/manifests/site.pp#L2207-L2242,` wdqs200[78]` are declared twice TASK DETAIL https://phabricator.wikimedia.org/T252068 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

RKemper added a comment. `wdqs1010` is one of our test servers Therefore can screw up `wdqs1010` as much as we want, but not the others --- we need to transfer that data to each server which implies we need to open a port for data transfer we have 2 primary DC: eqiad

[Wikidata-bugs] [Maniphest] [Created] T252068: WQDS Data Reload

RKemper created this task. RKemper added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION We have already done the reload on `wdqs1010`, so now we need to xfer to the various other