Re: [Wikitech-l] [Services] [ANNOUNCEMENT] RESTBase and related services DC switch-over test

2016-04-04 Thread Giuseppe Lavagetto
On Mon, Mar 14, 2016 at 10:54 PM, Marko Obrovac wrote: > Hello, > > The WMF’s technology department has for this quarter the goal of testing and > temporarily switching the main operational data centre from Eqiad (located > in Chicago) to Codfw (located in Dallas)~[1,2].

Re: [Wikitech-l] [Services] [ANNOUNCEMENT] RESTBase and related services DC switch-over test

2016-03-19 Thread Gabriel Wicke
The Services DC fail-over test finished without user impact, and traffic is now switched back to eqiad. We found an issue with one of the Cassandra nodes running out of memory after switching update processing to codfw, as well as a hick-up with one instance in eqiad. Due to the redundant set-up,