Hi, Many of you reported that various infra-owned services were not functioning properly these past two days. We had some package upgrades roll out that were bad, and a subset of services were non-functional while we worked through the incident.
(a) The upgrades did not appear to cause any data loss, but service availability was reduced (500s / 502s for some HTTP services.) (b) Some hosts may have had problems running some commands. E.g. I know that 'grep' didn't work for a while. If you observed errors related to missing 'GLIBC_2.33' symbols then this was likely related. (c) Many replicated services (including infra-status) rely on gitweb to take updates from git; and gitweb was not up, so they were unable to receive updates. (d) More impacted notes will follow in a postmortem that will come later this week for this incident. Thanks to all of you who reported problems and apologies for the service disruptions. -A