[Wikidata-bugs] [Maniphest] T354268: Wikidata-related Cloud VPS alerts about puppet

2024-01-03 Thread dcaro
dcaro added a comment. No problem, I'm here to help :}, my time is limited though, so might take some extra time for me to help on things "out of the usual" TASK DETAIL https://phabricator.wikimedia.org/T354268 EMAIL PREFERENCES https://phabricator.wikimedia.org/sett

[Wikidata-bugs] [Maniphest] T354268: Wikidata-related Cloud VPS alerts about puppet

2024-01-03 Thread dcaro
dcaro added a comment. You can probably change the default address pools in docker: # /etc/docker/daemon.json { "default-address-pools": [ {"base":"192.168.0.0/16","size":24} ] } But you might have to re

[Wikidata-bugs] [Maniphest] T354268: Wikidata-related Cloud VPS alerts about puppet

2024-01-03 Thread dcaro
dcaro added a comment. In T354268#9432766 <https://phabricator.wikimedia.org/T354268#9432766>, @AndrewTavis_WMDE wrote: > Thanks from me as well, @dcaro! Following up on my point: > >> Yep, those are yes, the networks they are configured to use (ex. 172.2

[Wikidata-bugs] [Maniphest] T354268: Wikidata-related Cloud VPS alerts about puppet

2024-01-03 Thread dcaro
dcaro added a comment. In T354268#9432608 <https://phabricator.wikimedia.org/T354268#9432608>, @Michael wrote: > In T354268#9432605 <https://phabricator.wikimedia.org/T354268#9432605>, @dcaro wrote: > >> The last alert, is because your puppetmaster `wdqspup

[Wikidata-bugs] [Maniphest] T354268: Wikidata-related Cloud VPS alerts about puppet

2024-01-03 Thread dcaro
dcaro added a comment. In T354268#9432641 <https://phabricator.wikimedia.org/T354268#9432641>, @AndrewTavis_WMDE wrote: > Hey @dcaro  Really appreciate you checking all of this for us! > >> So for that I guess the solution is to change your docker configuration t

[Wikidata-bugs] [Maniphest] T354268: Wikidata-related Cloud VPS alerts about puppet

2024-01-03 Thread dcaro
dcaro added a comment. The last alert, is because your puppetmaster `wdqspuppet` was not cleaned up when the VMs where removed, that has to be manually done: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates TASK DETAIL https

[Wikidata-bugs] [Maniphest] T354268: Wikidata-related Cloud VPS alerts about puppet

2024-01-03 Thread dcaro
dcaro added a comment. For `wikidata-analytics-1`, it seems that there's a docker installation there, that is reusing the same ranges that the cloud vps network uses, and the traffic to the DNS server is being sent through the docker interface: root@wikidata-analytics-1:/etc# grep

[Wikidata-bugs] [Maniphest] T354268: Wikidata-related Cloud VPS alerts about puppet

2024-01-03 Thread dcaro
dcaro added a comment. For the wikidata-federated-properties and wb-reconcile, the issue is that there's an old cron for the prometheus user: "/tmp/crontab.XOf4Il/crontab"

[Wikidata-bugs] [Maniphest] T329555: WDQS tutorial on toolforge is down

2023-10-27 Thread dcaro
dcaro closed subtask T329535: Cloud Ceph outage 2023-02-13 as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T329555 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lydia_Pintscher, dcaro Cc: WMDE-leszek, MarcoAurelio, Proeksad, Aklapper

[Wikidata-bugs] [Maniphest] T332953: Migrate PipelineLib repos to GitLab

2023-07-14 Thread dcaro
dcaro updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T332953 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: dcaro, kostajh, Gehel, jnuche, isarantopoulos, BPirkle, Tgr, Eevans, Seddon, MSantos, kevinbazira

[Wikidata-bugs] [Maniphest] T332953: Migrate PipelineLib repos to GitLab

2023-07-14 Thread dcaro
dcaro updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T332953 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: kostajh, Gehel, jnuche, isarantopoulos, BPirkle, Tgr, Eevans, Seddon, MSantos, kevinbazira

[Wikidata-bugs] [Maniphest] T265615: Terminating MySQL takes several minutes in (Wikibase?) CI jobs

2022-01-14 Thread dcaro
dcaro added a comment. I've added a burst configuration for 3x the iops for 10s for that flavor, might require stopping and starting (not restarting) the VMs: root@cloudvirt1024:~# openstack --os-project-id integration flavor show g2.cores8.ram24.disk80.4xiops

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2022-01-10 Thread dcaro
dcaro added a comment. If you create any new VMs, or re-image the existing ones, use that new flavor and it will avoid getting the host out of space (it has a slightly smaller disk). TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-15 Thread dcaro
dcaro added a comment. I had to stop (shutoff) and start the VM again (to clear the error state), the VM is up and running, and what's using all the space is: root@wcqs-beta-01:/srv/wdqs-data# du -hs * 4.0Kaliases.map 4.0Kdumps 28G latest-mediainfo.ttl.gz

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-14 Thread dcaro
dcaro added a comment. I've moved the other VM around, so there's a little bit of space free now, should be enough to start the VM but you'll have to make sure to cleanup right when it comes back up, if it's not possible to get it to less than half it's size, we will not be able to shrink

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-14 Thread dcaro
dcaro reopened this task as "Open". dcaro added a comment. Sorry about that. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: Base, RhinosF1, Majavah, aborrero, GFontenel

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-14 Thread dcaro
dcaro closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: Base, RhinosF1, Majavah, aborrero, GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominic

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-13 Thread dcaro
dcaro added a comment. The underlying filesystem is full, the disk size was set too big (3.4T), there's a cloudcanary instance taking 20G that can be removed to try to start the VM and cleanup stuff, but we'll need to resize the disk later once the sapce is freed. Not sure if that's

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-13 Thread dcaro
dcaro changed the task status from "In Progress" to "Open". TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: RhinosF1, Majavah, aborrero, GFontenelle_WMF, Sj,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-13 Thread dcaro
dcaro added a project: Cloud-Services-Origin-User. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD, Alicia_Fagerving_WMSE

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-13 Thread dcaro
dcaro added a project: Cloud-Services-Worktype-Unplanned. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-13 Thread dcaro
dcaro claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD, Alicia_Fagerving_WMSE, EBernhardson, Aklapper

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-13 Thread dcaro
dcaro changed the task status from "Open" to "In Progress". TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzhead

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2021-12-13 Thread dcaro
dcaro added a project: User-dcaro. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD, Alicia_Fagerving_WMSE, EBernhardson