Andrew,
I got 30+ emails and counting for stale file handles, permission errors,
email bounces, etc. Not sure what definition of brief you're using for
the outage (in my line of work that's sometimes less than a second), but
all these emails seem to have been send in a two hour window based on
the timestamps. Start 17:42 end 19:55 (both Amsterdam time). Maybe you
can stop cron next time you're going to planned break toollabs? Or was
this caused by the dns + network outage?
Maarten
On 07-10-19 20:55, Andrew Bogott wrote:
This took a lot longer than expected, but everything is now upgraded.
There was a brief dns + network outage throughout the cloud which also
affected toolforge. We believe that all of those issues are resolved
now, but please let us know if you see any bad long-term effects.
On 10/1/19 1:24 PM, Andrew Bogott wrote:
We'll be upgrading the cloud services OpenStack install on Monday,
beginning at 14:00 UTC.
The entire upgrade process may take a couple of hours. Early on in
the process, Horizon (and associated OpenStack APIs) will be disabled
(probably for 20 to 30 minutes.) There may also be brief network
interruptions during the upgrade, although if all goes well these
will not be noticeable by users.
Toolforge and existing VMs should be largely unaffected apart from
possible network hiccups.
- Andrew + the WMCS team
_______________________________________________
Wikimedia Cloud Services announce mailing list
[email protected] (formerly
[email protected])
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
_______________________________________________
Wikimedia Cloud Services mailing list
[email protected] (formerly [email protected])
https://lists.wikimedia.org/mailman/listinfo/cloud
_______________________________________________
Wikimedia Cloud Services mailing list
[email protected] (formerly [email protected])
https://lists.wikimedia.org/mailman/listinfo/cloud
