On Wed, Oct 18, 2017 at 6:57 PM, John <[email protected]> wrote: > Would it not be better to trial the switch over with the *.labsdb aliases > before risking catastrophic failure when rebooting? Doing a short test (up > to 24 hours) would allow users to identify anything that may break before it > becomes an unbreak now situation? If any critical systems are affected the > change can be rolled back, issues identified and fixed before the final > switch over.
Switching the "c1.labsdb" and "c3.labsdb" aliases will break 100% of the Tools and other users that are using user created databases on the hosts they currently point to. The server reboots have a small chance of non-recoverable hardware failure, but there is really nothing we can do to prevent that. We considered putting the reboots off and just waiting for the servers to be decommissioned, but ultimately it was decided that the risk of continuing to run out of date Linux kernels was worse than the risk of hardware failure. There is a tool at <https://tools.wmflabs.org/tool-db-usage/> where you can see the local databases on the c1 and c3 servers. Anything that is using these tables will need to change behavior somehow. It is not a happy thing for us to force anyone to change their software, but as explained in the wiki page [0] we can not find a reliable method to ensure that the same user created tables are available on all three of the new backend servers, and we feel that in order to be able to properly maintain the new servers we need to have more flexibility in choosing which traffic goes to which physical server at any given point in time. If we recreate the current state where certain service names are guaranteed to point to specific physical servers Tools will break at arbitrary times when we are doing otherwise invisible maintenance. [0]: https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown Bryan -- Bryan Davis Wikimedia Foundation <[email protected]> [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Wikimedia Cloud Services mailing list [email protected] (formerly [email protected]) https://lists.wikimedia.org/mailman/listinfo/cloud
