Michael created this task. Michael added projects: Wikidata, Cloud-Services-Origin-Alert, cloud-services-team, Wikibase - Federated Properties, WMDE-Analytics-Engineering. Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION We are seeing some ongoing warnings on https://alerts.wikimedia.org that seem to be related to Wikidata/WMDE projects on Clouds-VPS In particular, we are seeing the following types of warnings: **PuppetAgentNoResources** - No Puppet resources found on instance wb-reconcile on project wikidata-dev - No Puppet resources found on instance wikidata-analytics-1 on project wmdeanalytics **PuppetAgentStaleLastRun** - Last Puppet run was over 24 hours ago on instance wikidata-federated-properties in project wikidata-dev - Last Puppet run was over 24 hours ago on instance wb-reconcile in project wikidata-dev **PuppetStaleCertificates** - Found non-revoked Puppet certificates for 3 deleted instances on wdqspuppet See the current state at: https://alerts.wikimedia.org/?q=%40state%3Dactive&q=wikidata&q=%40cluster%3Dwmcloud.org Looking into this with @dcaro in the `#wikimedia-cloud` IRC channel (logs <https://wm-bot.wmcloud.org/browser/index.php?start=01%2F03%2F2024&end=01%2F03%2F2024&display=%23wikimedia-cloud>), it seems that there is some permissions issue causing the `PuppetAgentNoResources` error. See the chat-logs for more detail. That issue might also affect other projects that show the same warning (prometheus query which is linked as the warning's source <https://prometheus.wmcloud.org/graph?g0.expr=puppet_agent_resources_total%20%3D%3D%200&g0.tab=1&g0.stacked=0&g0.range_input=4w&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D>). Also, there seems to be an DNS issue with the `wikidata-analytics-1` project. It is running the `quratorqcerevolver` and `quratorqcfrevolver` containers. Rebooting it to force a network refresh did not help. (SAL <https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmdeanalytics/SAL>) **Notes** - Independently, that federated-properties wiki seems broken: https://wikidata-federated-properties.wmflabs.org/ due to a PHP/MediaWiki mismatch. That has been highlighted to the respective teams at WMDE to decide how to move forward with that. TASK DETAIL https://phabricator.wikimedia.org/T354268 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael Cc: Aklapper, Manuel, Andrew-WMDE, Michael, dcaro, Danny_Benjafield_WMDE, VeniVidiVicipedia, Astuthiodit_1, karapayneWMDE, Invadibot, Devnull, Slst2020, maantietaja, nskaggs, Mohammed_Sadat_WMDE, Raymond_Ndibe, ItamarWMDE, Akuckartz, Nandana, sietec, lucamauri, Lahi, Gq86, GoranSMilovanovic, Chicocvenancio, QZanden, LawExplorer, Regisrob, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, bd808
_______________________________________________ Wikidata-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected]
