[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-01-24T22:54:13Z] T280001 Removed downtime on `wcqs*` TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-01-24T22:48:19Z] T280001 Moved `wcqs` service state into `production` by merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/756713; running puppet on

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 756713 **merged** by Bking: [operations/puppet@production] wcqs: move service into production status https://gerrit.wikimedia.org/r/756713 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 756713 had a related patch set uploaded (by Bking; author: Bking): [operations/puppet@production] wcqs: move service into production status https://gerrit.wikimedia.org/r/756713 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-01-24T22:32:24Z] T280001 T282117 Merged https://gerrit.wikimedia.org/r/c/operations/dns/+/755806 and ran `sudo -i authdns

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 755806 **merged** by Bking: [operations/dns@master] wcqs: add discovery record https://gerrit.wikimedia.org/r/755806 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 755810 **merged** by Bking: [operations/puppet@production] wcqs: Move back from lvs_setup to monitoring_setup https://gerrit.wikimedia.org/r/755810 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 755810 had a related patch set uploaded (by Bking; author: Bking): [operations/puppet@production] wcqs: Move back from lvs_setup to monitoring_setup https://gerrit.wikimedia.org/r/755810 TASK DETAIL https://phabricator.wikimedia.org/T280001

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 755806 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/dns@master] wcqs: add discovery record https://gerrit.wikimedia.org/r/755806 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:21:06Z] T280001 Rolling restart of low-traffic pybal hosts complete. All of `wcqs` is pooled and the pybal / ipvs related alerts have cleared TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:16:45Z] T280001 Pooled `wcqs200[1-3]` (had been left unpooled from when we last removed wcqs from production) TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:12:17Z] T280001 Restarting pybal on low-traffic primaries `lvs2009` and `lvs1015`: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:12:07Z] T280001 Restarting pybal on low-traffic primaries `lvs2009` and `lvs1015`: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:11:30Z] T280001 Waited 120s and checked https://icinga.wikimedia.org/alerts, proceeding to primary low-traffic hosts `lvs2009` and `lvs1015` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:08:32Z] T280001 Sanity check of `sudo ipvsadm -L -n` on backup `lvs2010` and `lvs1016` looks good (for ex `lvs1016` has `TCP 10.2.2.67:443 wrr`) TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:07:45Z] T280001 Restarting pybal on low-traffic backups: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2010*,lvs1016*}' 'sudo systemctl restart pybal'` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:02:53Z] T280001 `ryankemper@cumin1001:~$ sudo cumin 'O:lvs::balancer' 'sudo run-puppet-agent'` TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:01:47Z] T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/742841 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 742841 **merged** by Ryan Kemper: [operations/puppet@production] wcqs: move back to lvs_setup https://gerrit.wikimedia.org/r/742841 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-02T01:00:40Z] T280001 About to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/742841 to bring `wcqs` into state `lvs_setup`, after which I'll perform a

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 742841 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wcqs: move back to lvs_setup https://gerrit.wikimedia.org/r/742841 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF,

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 724545 **merged** by Ryan Kemper: [operations/puppet@production] wcqs: add disc desired state https://gerrit.wikimedia.org/r/724545 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 724545 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wcqs: add disc desired state https://gerrit.wikimedia.org/r/724545 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-29T00:21:18Z] T280001 `ryankemper@authdns1001:~$ sudo -i authdns-update` following merge of https://gerrit.wikimedia.org/r/c/operations/dns/+/724538 TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-29T00:19:32Z] T280001 Okay now we're clear to proceed to https://wikitech.wikimedia.org/wiki/LVS#For_active/active_services; merging

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-29T00:15:05Z] T280001 `ryankemper@cumin1001:~$ sudo cumin 'A:icinga or A:dns-auth' run-puppet-agent` per

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-29T00:14:34Z] T280001 Moving wcqs state from `monitoring_setup` to `production`; merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/724536 TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 724536 **merged** by Ryan Kemper: [operations/puppet@production] wcqs: state change: monitoring_setup -> production https://gerrit.wikimedia.org/r/724536 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 724536 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wcqs: state change: monitoring_setup -> production https://gerrit.wikimedia.org/r/724536 TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-28T23:53:42Z] T280001 New icinga checks are green, will proceed to next step of moving wcqs state from `monitoring_setup` -> `production` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-28T23:53:05Z] T280001 New icinga checks are green, will proceed to next step of moving wcqs state from `monitoring_setup` -> `production_setup` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-28T23:49:52Z] T280001 New icinga alerts showing up as expected following wcqs state change to `monitoring_setup`: `LVS wcqs codfw port 443/tcp - Wikimedia Commons Query

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-28T23:45:15Z] T280001 Changing wcqs state from `lvs_setup` to `monitoring_setup`: `ryankemper@cumin1001:~$ sudo cumin 'A:icinga' 'run-puppet-agent'` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 724533 **merged** by Ryan Kemper: [operations/puppet@production] wcqs: state change: lvs_setup -> monitoring_setup https://gerrit.wikimedia.org/r/724533 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 724533 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wcqs: state change: lvs_setup -> monitoring_setup https://gerrit.wikimedia.org/r/724533 TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-28T18:54:19Z] T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/721600 (add wcqs scap dsh groups), running puppet on scap::dsh hosts:

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 721600 **merged** by Ryan Kemper: [operations/puppet@production] Add dsh targets for the new wcqs cluster https://gerrit.wikimedia.org/r/721600 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 721600 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] Add dsh targets for the new wcqs cluster https://gerrit.wikimedia.org/r/721600 TASK DETAIL https://phabricator.wikimedia.org/T280001

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 723314 **merged** by Ryan Kemper: [operations/puppet@production] query_service: Add monitoring::groups for wcqs https://gerrit.wikimedia.org/r/723314 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 723314 had a related patch set uploaded (by Ryan Kemper; author: Ebernhardson): [operations/puppet@production] query_service: Add monitoring::groups for wcqs https://gerrit.wikimedia.org/r/723314 TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:27:50Z] T280001 `ryankemper@cumin1001:~$ sudo cumin 'P{puppetmaster*}' 'sudo rm -fv /var/run/confd-template/.wcqs*'` complete, forcing recheck TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:27:01Z] T280001 The pooling of the `wcqs*` hosts has gotten `/srv/config-master/pybal/${DC}/wcqs` to render, but we need to clear away the stale error files to get

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:18:21Z] T280001 `ryankemper@puppetmaster1001:/srv$ sudo confctl select 'name=wcqs.*' set/pooled=yes:weight=10` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:13:17Z] T280001 [codfw] `root@lvs2010:/home/ryankemper# ipvsadm -Dt 10.2.2.67:443` and `root@lvs2009:/home/ryankemper# ipvsadm -Dt 10.2.2.67:443` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:13:10Z] T280001 [eqiad] `root@lvs1016:/home/ryankemper# ipvsadm -Dt 10.2.1.67:443` and `root@lvs1015:/home/ryankemper# ipvsadm -Dt 10.2.1.67:443` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:06:54Z] T280001 Restarted pybal on low-traffic primaries: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl restart pybal'` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:06:33Z] T280001 Waited 120s and checked https://icinga.wikimedia.org/alerts, proceeding to primary low-traffic hosts `lvs2009` and `lvs1015` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:05:50Z] T280001 [Cleanup required] `TCP 10.2.1.67:443 wrr` shows up on `ryankemper@lvs1016:~$ sudo ipvsadm -L -n` and `TCP 10.2.2.67:443 wrr` shows up on

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:05:32Z] T280001 [Sanity check] `TCP 10.2.2.67:443 wrr` shows up on `ryankemper@lvs1016:~$ sudo ipvsadm -L -n` and `TCP 10.2.1.67:443 wrr` shows up on

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:04:09Z] T280001 Restarted pybal on low-traffic backups: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2010*,lvs1016*}' 'sudo systemctl restart pybal'` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:03:51Z] T280001 Restarting pybal on low-traffic backups `lvs2010` and `lvs1016`... TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:03:18Z] T280001 Ran puppet on all lvs hosts: `ryankemper@cumin1001:~$ sudo cumin 'O:lvs::balancer' 'sudo run-puppet-agent'` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T22:00:52Z] T280001 Running puppet on all lvs hosts: `ryankemper@cumin1001:~$ sudo cumin 'O:lvs::balancer' 'sudo run-puppet-agent'`... TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:59:49Z] T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/723315, ran puppet agent on `wcqs*` to fix `local lo:LVS destination IPs` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:59:08Z] T280001 Swapped the netbox IPAM addresses back, after erroneously swapping them earlier. `sre.dns.netbox` cookbook run complete as well TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 723315 **merged** by Ryan Kemper: [operations/puppet@production] wcqs: fix swapped codfw / eqiad ip defaults https://gerrit.wikimedia.org/r/723315 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 723315 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wcqs: fix swapped codfw / eqiad ip defaults https://gerrit.wikimedia.org/r/723315 TASK DETAIL https://phabricator.wikimedia.org/T280001

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:36:04Z] T280001 `sre.dns.netbox` run complete, netbox IP mixup *should* be resolved TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:26:59Z] T280001 `ryankemper@cumin1001:~$ sudo -i cookbook sre.dns.netbox -t T280001 'Fix swapped wcqs.svc.[eqiad,codfw].wmnet'` in progress (note: no `sudo

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:23:28Z] T280001 Swapped IPs of https://netbox.wikimedia.org/ipam/ip-addresses/9062/ and https://netbox.wikimedia.org/ipam/ip-addresses/9063; this should fix the

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:05:44Z] T280001 Restarted pybal on low-traffic primaries: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl restart pybal'` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:04:40Z] T280001 Restarting pybal on low-traffic primaries `lvs2009` and `lvs1015`... TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:04:12Z] T280001 Waited 120s and checked https://icinga.wikimedia.org/alerts, proceeding to primary low-traffic hosts `lvs2009` and `lvs1015` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:00:55Z] T280001 Sanity check of `sudo ipvsadm -L -n` on low-traffic backups `lvs2010` and `lvs1016` looks good, proceeding TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:00:14Z] T280001 Sanity check of `sudo ipvsadm -L -n` on backup `lvs2010` and `lvs1016` looks good, proceeding TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T21:00:05Z] T280001 `TCP 10.2.1.67:443 wrr` shows up on `ryankemper@lvs1016:~$ sudo ipvsadm -L -n ` and `TCP 10.2.2.67:443 wrr` shows up on `ryankemper@lvs2010:~$

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T20:54:37Z] T280001 Restarted pybal on backup low-traffic hosts: `ryankemper@cumin1001:~$ sudo cumin 'P{lvs2010*,lvs1016*}' 'sudo systemctl restart pybal'` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T20:53:55Z] T280001 Restarting pybal on backup low-traffic hosts `lvs2010` and `lvs1016`... TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T20:53:10Z] T280001 Ran puppet on all lvs hosts => `ryankemper@cumin1001:~$ sudo cumin 'O:lvs::balancer' 'sudo run-puppet-agent'` TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 723254 **merged** by Ryan Kemper: [operations/puppet@production] wcqs: go from service_setup to lvs_setup https://gerrit.wikimedia.org/r/723254 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T20:47:23Z] T280001 Merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/723254 to proceed with `lvs_setup` state change; will be restarting low-traffic lvs

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

RKemper added a comment. [Steps for deploy of https://gerrit.wikimedia.org/r/723254] --- merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/723254 (on cumin) => sudo cumin 'O:lvs::balancer' 'sudo run-puppet-agent'

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T18:58:37Z] T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/721089 to see if it resolves the `confd` error that popped up TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 721089 **merged** by Ryan Kemper: [operations/puppet@production] Declare wikimedia_cluster for wcqs https://gerrit.wikimedia.org/r/721089 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 721089 had a related patch set uploaded (by Ryan Kemper; author: Ebernhardson): [operations/puppet@production] Declare wikimedia_cluster for wcqs https://gerrit.wikimedia.org/r/721089 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 723254 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wcqs: go from service_setup to lvs_setup https://gerrit.wikimedia.org/r/723254 TASK DETAIL https://phabricator.wikimedia.org/T280001

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

RKemper added a comment. [conftool changes following merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/713959] - Now running conftool-merge to sync any changes to conftool

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-23T16:38:46Z] T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/713959, running puppet on `*w*qs*` (i.e. wcqs and wdqs) TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 713959 **merged** by Ryan Kemper: [operations/puppet@production] query_service: LVS for WCQS step 1 https://gerrit.wikimedia.org/r/713959 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 713959 had a related patch set uploaded (by Ryan Kemper; author: Ebernhardson): [operations/puppet@production] blazegraph: LVS for WCQS step 1 https://gerrit.wikimedia.org/r/713959 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

RKemper added a comment. [DNS change -> sre.dns.netbox cookbook] --- ryankemper@cumin1001:~$ sudo -i cookbook sre.dns.netbox -t T280001 'Added wcqs.svc.[eqiad,codfw].wmnet' START - Cookbook sre.dns.netbox Generating the DNS records from

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-21T19:10:43Z] T280001 `sre.dns.netbox` completed successfully TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-21T18:56:55Z] T280001 Running `sudo -i cookbook sre.dns.netbox -t T280001 'Added wcqs.svc.[eqiad,codfw].wmnet'` per final step of

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-21T18:53:18Z] T280001 `for i in 0 1 2 ; do dig @ns${i}.wikimedia.org -t any wcqs.svc.[eqiad,codfw].wmnet ; done` looks as expected TASK DETAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

RKemper added a comment. [DNS change -> eqiad validation] ryankemper@authdns1001:~$ for i in 0 1 2 ; do dig @ns${i}.wikimedia.org -t any wcqs.svc.eqiad.wmnet ; done ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns0.wikimedia.org -t any

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-21T18:48:08Z] T280001 `OK - authdns-update successful on all nodes!` TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-21T18:45:31Z] T280001 `ryankemper@authdns1001:~$ sudo authdns-update` TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-21T18:44:58Z] T280001 Merging https://gerrit.wikimedia.org/r/c/operations/dns/+/713929; will follow steps in

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 713929 **merged** by Ryan Kemper: [operations/dns@master] Add wcqs.svc.{codfw,eqiad}.wmnet https://gerrit.wikimedia.org/r/713929 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 713958 **merged** by Ryan Kemper: [operations/puppet@production] blazegraph: Setup tls termination for wcqs https://gerrit.wikimedia.org/r/713958 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

RKemper added a comment. Update: `sdcquery01` has been deleted, so we can proceed with removing the `Optional` type TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: EBernhardson,

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

RKemper added a comment. With respect to https://gerrit.wikimedia.org/r/c/operations/puppet/+/713946, we'll need to know if `sdcquery01` is still in use. If it's not and we can "decommission" it (or at least not worry about breaking it), that will let us require `oauth_settings` to be

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 715569 **merged** by Ryan Kemper: [operations/puppet@production] wcqs: create tls cert https://gerrit.wikimedia.org/r/715569 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 715570 **merged** by Ryan Kemper: [labs/private@master] wcqs: add wcqs.discovery.wmnet dummy key https://gerrit.wikimedia.org/r/715570 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

RKemper added a comment. Log of steps to generate tls certs for wcqs with cergen for https://gerrit.wikimedia.org/r/713958 - `/srv/private` commit + 2 gerrit patches for TLS certs:

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 715570 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [labs/private@master] wcqs: add wcqs.discovery.wmnet dummy key https://gerrit.wikimedia.org/r/715570 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 715569 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wcqs: create tls cert https://gerrit.wikimedia.org/r/715569 TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

EBernhardson reassigned this task from EBernhardson to RKemper. EBernhardson added a comment. The initial patches hopefully help a bit, but there are a number of manual tasks requiring ops access to move this forward any further. TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

gerritbot added a comment. Change 713958 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [operations/puppet@production] blazegraph: Setup tls termination for wcqs https://gerrit.wikimedia.org/r/713958 TASK DETAIL https://phabricator.wikimedia.org/T280001

  1   2   >