RKemper added a comment.
Note: Puppet is still disabled on `wdqs2008` while the reload runs. It
occurred to me that I'm not sure if puppet actually **needs** to be disabled
during data reloads or if that's just a precaution we've historically taken -
any insight here @Gehel?
TASK DETAIL
RKemper added a comment.
@Cmjohnson The data reload is complete on `wdqs1009`, so the host can now
have its firmware upgraded and be rebooted at its convenience. Note this is an
internal wdqs test host, so there is no public-facing service for us to worry
about.
Feel free to proceed
RKemper added a comment.
Downtimed `wdqs2008` until `2021-03-04 21:56:59`
TASK DETAIL
https://phabricator.wikimedia.org/T267927
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Zbyszko, Gehel, Aklapper, dcausse, MPhamWMF, CBogen
RKemper added a comment.
Still waiting for the latest dumps to be downloaded (few more hours), then
need to reboot WDQS hosts as part of https://phabricator.wikimedia.org/T274213,
then can do the actual data-reload
TASK DETAIL
https://phabricator.wikimedia.org/T267927
EMAIL PREFERENCES
RKemper added a comment.
(See https://phabricator.wikimedia.org/T273097#6805355 for why this ticket
has been closed)
TASK DETAIL
https://phabricator.wikimedia.org/T266495
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: RKemper
RKemper added a comment.
Updated:
[X] https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service#Updates
[X]
https://commons.wikimedia.org/wiki/Commons:Village_pump#Unscheduled_maintenance%3A_Wikimedia_Commons_Query_Service
[X] WikiData ML mailing list (note: A community member
RKemper moved this task from In Progress to Needs Reporting on the
Discovery-Search (Current work) board.
RKemper added a comment.
WCQS is back in service; updating the notification channels right now and
will comment back here after
TASK DETAIL
https://phabricator.wikimedia.org/T273636
RKemper claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T273636
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana,
Namenlos314, Lahi, Gq86
RKemper added a comment.
`sudo cookbook sre.wdqs.data-reload wdqs1009.eqiad.wmnet
--reuse-downloaded-dump --reload-data wikidata --skolemize --reason 'T267927:
Reload wikidata jnl from fresh dumps' --task-id T267927 ` is failing with:
- OUTPUT of 'test -f /srv/wdq...test
RKemper added a comment.
TODO from IRC meeting with bblack/gehel: create a DNS entry (CNAME to
dyna.wm.o), another set of entries in backend.yaml map, create another minisite
(with the appropriate configuration)
TASK DETAIL
https://phabricator.wikimedia.org/T266470
EMAIL PREFERENCES
RKemper added a comment.
Notified WikiData mailing list and also posted here:
https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service#Updates
TASK DETAIL
https://phabricator.wikimedia.org/T273636
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
RKemper added a comment.
`wcqs-beta-01.eqiad.wmflabs` is running low on disk space due to its
blazegraph journal dataset size. In order to free up space we will need to take
the service down, delete the journal and re-import from the latest dump.
Service interruption will begin at Feb 4 18
RKemper added a comment.
@akosiaris Is your concern with the idea of using a`flink` base image
solution mainly just centered around the inefficiency/inconvenience of needing
SRE to merge any flink version upgrades? Since we have an embedded SRE on
search (me) and to a lesser extent
RKemper added a comment.
We'll want to reload these this Friday, because the latest dumps should be
available thursday evening.
TASK DETAIL
https://phabricator.wikimedia.org/T267927
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc
RKemper claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T267927
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi,
Gq86, Lucas_Werkmeister_WMDE
RKemper triaged this task as "Medium" priority.
RKemper claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T266470
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamW
RKemper updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T273097
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: dcausse, Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, CBogen, Akuckartz,
Nandana, Namenlos314
RKemper added a comment.
In T271851#6765207 <https://phabricator.wikimedia.org/T271851#6765207>,
@Ladsgroup wrote:
> @RKemper I'm so sorry it was buggy, the cleanest way forward here is to
deploy wcqs using microsites and we can easily drop everything from puppet
then. Does
RKemper added a comment.
Post deploy check:
`wdqs-updater.service` is now running without the `--import-async` flag as
expected:
`ExecStart=/bin/bash /srv/deployment/wdqs/wdqs/runUpdate.sh -n wdq -- --kafka
kafka-main1001.eqiad.wmnet:9092,kafka-main1002.eqiad.wmnet:9092,kafka
RKemper added a comment.
(Puppet run successful on wdqs nodes following deploy)
TASK DETAIL
https://phabricator.wikimedia.org/T267175
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Skim, Strepon, Multichill, Zbyszko, RKemper
RKemper moved this task from In Progress to To Be Deployed on the
Discovery-Search (Current work) board.
RKemper added a comment.
Deploying https://gerrit.wikimedia.org/r/656833 now
TASK DETAIL
https://phabricator.wikimedia.org/T267175
WORKBOARD
https://phabricator.wikimedia.org/project
RKemper added a comment.
I responded to Maciej (the one who reported the problem originally) on the
mailing list. Looks like we accidentally gave them the wrong ticket number
earlier: https://lists.wikimedia.org/pipermail/wikidata/2021-January/014451.html
Looks like this ticket hasn't
RKemper updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T271412
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Aklapper, Zbyszko, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi,
Gq86
RKemper moved this task from In Progress to Needs Reporting on the
Discovery-Search (Current work) board.
RKemper added a comment.
This is all done. No issues currently.
TASK DETAIL
https://phabricator.wikimedia.org/T244753
WORKBOARD
https://phabricator.wikimedia.org/project/board/1227
RKemper triaged this task as "High" priority.
TASK DETAIL
https://phabricator.wikimedia.org/T270236
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Aklapper, RKemper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, L
RKemper added a project: Wikidata-Query-Service.
Restricted Application added a project: Wikidata.
TASK DETAIL
https://phabricator.wikimedia.org/T270236
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Aklapper, RKemper, MPhamWMF, CBogen
RKemper moved this task from In Progress to Needs Reporting on the
Discovery-Search (Current work) board.
RKemper added a comment.
The above two patches (really one patch with a followup patch to fix a typo)
seem to have fixed the problem. I'll want to circle back to verify we're
getting
RKemper added a comment.
Restarting `prometheus-blazegraph-exporter-wdqs-blazegraph.service` after
switching from Counter to Gauge now shows the correct `blazegraph_lastupdated`
metric when running `curl localhost:9193` (9193 is the port for
wdqs-blazegraph).
Still waiting to see
RKemper renamed this task from "wdqs-categories prometheus exporter failing on
wdqs1011" to "wdqs-categories prometheus exporter failing on select wdqs
instances".
RKemper updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T269872
EMAIL
RKemper created this task.
RKemper added projects: Wikidata-Query-Service, Discovery-Search (Current work).
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
ryankemper@wdqs1011:~$ sudo systemctl status
prometheus
RKemper added a comment.
Job lives here:
https://github.com/wikimedia/mediawiki-extensions-Wikidata.org/blob/60c5f96ebf424b792077bb7c6b533a68702e7aea/maintenance/updateQueryServiceLag.php#L70
I have a patch open here:
https://gerrit.wikimedia.org/r/c/operations/puppet/+/646888
RKemper added a comment.
There's some context in the description of
https://phabricator.wikimedia.org/T269204 that mentions that the counter metric
`blazegraph_lastupdated` is now `blazegraph_lastupdated_total`, so if
the`mediawiki_job_wikidata-updateQueryServiceLag` job has to do
RKemper moved this task from In Progress to Needs Reporting on the
Discovery-Search (Current work) board.
RKemper added a comment.
All categories reloads/transfers are complete
TASK DETAIL
https://phabricator.wikimedia.org/T259588
WORKBOARD
https://phabricator.wikimedia.org/project
RKemper updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T269302
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Lydia_Pintscher, Vojtech.dostal, Dipsacus_fullonum, Aklapper, Muchiri124,
CBogen, Nintendofan885
RKemper added a comment.
Here's something interesting: On a server where I've recently run the
`data-reload` for `categories` (`wdqs1006` in this case), the categories
journal looks like so:
`-rw-rw-r-- 1 blazegraph blazegraph 21G Dec 3 07:19 categories.jnl`
Yet on a wdqs
RKemper added a comment.
Note: https://phabricator.wikimedia.org/T269331 was created due to an
exception encountered every time we run the data reload cookbook. The exception
only occurs at the very end, so it might not indicate an actual problem
TASK DETAIL
https
RKemper claimed this task.
RKemper updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T269204
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Gehel, RKemper, dcausse, Aklapper, lmata, CBogen, Akuckartz, Nandana
RKemper added a comment.
`sudo -i wmf-auto-reimage-host --conftool -p T269204 wdqs2004.codfw.wmnet` is
an example of how to reimage hosts (run from cumin)
TASK DETAIL
https://phabricator.wikimedia.org/T269204
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
RKemper moved this task from Ready for Development to In Progress on the
Discovery-Search (Current work) board.
RKemper added a comment.
Will start on this today
TASK DETAIL
https://phabricator.wikimedia.org/T259588
WORKBOARD
https://phabricator.wikimedia.org/project/board/1227/
EMAIL
RKemper moved this task from In Progress to Needs Reporting on the
Discovery-Search (Current work) board.
RKemper added a comment.
We need to do another reload but not under this ticket
TASK DETAIL
https://phabricator.wikimedia.org/T255399
WORKBOARD
https://phabricator.wikimedia.org
RKemper moved this task from In Progress to Needs Reporting on the
Discovery-Search (Current work) board.
RKemper added a comment.
This should be resolved now; the transfer is complete and the node matches
others in the fleet.
ryankemper@wdqs2002:~$ ls -lh /srv/wdqs/wikidata.jnl
RKemper updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T262009
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Aklapper, RKemper, Gehel, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlos314, Lahi, Gq86
RKemper moved this task from Needs review to Needs Reporting on the
Discovery-Search (Current work) board.
RKemper added a comment.
Applied this today
TASK DETAIL
https://phabricator.wikimedia.org/T258835
WORKBOARD
https://phabricator.wikimedia.org/project/board/1227/
EMAIL PREFERENCES
RKemper moved this task from Needs review to Needs Reporting on the
Discovery-Search (Current work) board.
RKemper added a comment.
We'll want to monitor across the next week to verify all looks good.
TASK DETAIL
https://phabricator.wikimedia.org/T261204
WORKBOARD
https
RKemper added a comment.
dumpsgen 58563 0.0 0.0 4276 700 ?Ss 20:50 0:00 /bin/sh
-c python3 /srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py
--configfile /etc/dumps/confs/addschanges.conf --dumptype incrdumps --quiet
dumpsgen 58565 0.0 0.0 57976
RKemper added a comment.
We discussed this during this week's SRE meeting and resolved to enable full
root access for wdqs admins, rather than granularly expanding access one file
at a time. This also lines up better with how we currently manage
Elasticsearch, where admins have root access
RKemper added a project: Sustainability (Incident Followup).
TASK DETAIL
https://phabricator.wikimedia.org/T258739
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: RKemper, CDanis, dcausse, Aklapper, Dzahn, lmata, Alter-paule, Beast1978
RKemper renamed this task from "wdqs admins should have access to nginx logs on
wdqs machines" to "wdqs admins should have access to nginx logs, jstack on wdqs
machines".
TASK DETAIL
https://phabricator.wikimedia.org/T258739
EMAIL PREFERENCES
https://phabricator.wi
RKemper closed subtask T252068: WQDS Data Reload as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T230588
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko, RKemper
Cc: Ghuron, Nikki, Toni_001, Dipsacus_fullonum, Larske, Mathew.onipe
RKemper closed subtask T252068: WQDS Data Reload as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T240831
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Gehel, Aklapper, Mbch331, CBogen, darthmon_wmde, Nandana, Lahi, Gq86
RKemper closed this task as "Resolved".
RKemper added a comment.
Still need to circle back to dependent tickets and verify that those problems
are solved
TASK DETAIL
https://phabricator.wikimedia.org/T252068
EMAIL PREFERENCES
https://phabricator.wikimedia.org/sett
RKemper closed subtask T252068: WQDS Data Reload as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T245135
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Lucas_Werkmeister_WMDE, Aklapper, Dipsacus_fullonum, CBogen, darthmon_wmde
RKemper added a comment.
Data transfer is done across all instances as of last friday, along with a
wdqs-categories reload that we tacked on.
Circling back to mark this ticket as done.
TASK DETAIL
https://phabricator.wikimedia.org/T252068
EMAIL PREFERENCES
https
RKemper added a comment.
should also circle back and fix
https://github.com/wikimedia/puppet/blob/production/manifests/site.pp#L2207-L2242,`
wdqs200[78]` are declared twice
TASK DETAIL
https://phabricator.wikimedia.org/T252068
EMAIL PREFERENCES
https://phabricator.wikimedia.org
RKemper added a comment.
`wdqs1010` is one of our test servers
Therefore can screw up `wdqs1010` as much as we want, but not the others
---
we need to transfer that data to each server
which implies we need to open a port for data transfer
we have 2 primary DC: eqiad
RKemper created this task.
RKemper added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
We have already done the reload on `wdqs1010`, so now we need to xfer to the
various other
56 matches
Mail list logo
