Hello! I just finished writing a short incident report on the maps issue of last Friday [1].
Basically, I was stupid and reinitialized Cassandra on the wrong node. This should have had almost no effect, except that the system_auth keyspace of Cassandra is configured with a replication factor of 1. Loosing that node means that we also lost authentication. Thanks a lot to Brandon for the help in mitigating this! [1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20161021-Maps -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST _______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
