Re: [Wikitech-l] Phabricator outage due to network issues, 11/29
Hi, let me recycle this reply posted initially at Determine phabricator.wikimedia.org service level - https://phabricator.wikimedia.org/T76381 Currently Phabricator is getting the same service level that Bugzilla had. Looking at the whole Wikimedia picture, I think this is the most sensible option. I don't see any strong reason to change it. Bugzilla was down unexpectedly several times in the past years, and if Ops was able to react quicker it's just because we were luckier with the cause, timing and location of the breaks. If we would have Bugzilla instead of Phabricator in the rack that went down this weekend, the service provided by Ops would have been exactly the same. We can reopen this discussion when planning the migration of code review and (eventually) continuous integration. For now, I think we are good. This is the opinion of the Engineering Community team. If this works also for Operations and Platform Engineering, then we can resolve this task. PS: About the downtime itself, 5 hours on a weekend is clearly unfortunate, but imho nothing that should make us revise the current service level. Was anybody unable to work, arms crossed? Was any project delayed? I'm counting volunteers as much as employees. Personally I learned about the downtime only in wikitech-l, having used Phabricator on Saturday-Sunday night at 1am CET, and then on Sunday at 1pm. -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Phabricator outage due to network issues, 11/29
As noted in the server admin log [1], Phabricator is currently down due to a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP. Erik [1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log -- Erik Möller VP of Product Strategy, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Phabricator outage due to network issues, 11/29
ASAP? when it's already hitting approx. five hours of down time? On 30 November 2014 at 18:14, Erik Moeller e...@wikimedia.org wrote: As noted in the server admin log [1], Phabricator is currently down due to a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP. Erik [1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log -- Erik Möller VP of Product Strategy, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Phabricator outage due to network issues, 11/29
Hoi, Right ? so it is thanksgiving et al.. Be thankful that it is seen, It is not Wikipedia or any of the projects...so relax.. eat some left over turkey.. Thanks, GerardM On 30 November 2014 at 09:59, K. Peachey p858sn...@gmail.com wrote: ASAP? when it's already hitting approx. five hours of down time? On 30 November 2014 at 18:14, Erik Moeller e...@wikimedia.org wrote: As noted in the server admin log [1], Phabricator is currently down due to a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP. Erik [1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log -- Erik Möller VP of Product Strategy, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Phabricator outage due to network issues, 11/29
This last line in the conversation strikes me as dry, useless and submissive. We should not ever neglect a sister project (including wikimedia projects, phabricator, wikitech, tools, etc), as small as their userbase might seem. For many, weekends are the volunteering or coding time and if the website they use for it was off, such people would be frustrated. Would be interesting to see how to set up multi-server instance of FAB. http://blog.iweb.com/en/2012/02/how-to-distribute-website-load-across-multiple-servers/9867.html What I don't understand is how to decentralise the database. Happy Thanksgiving to all, of course... -- svetlana On Sun, 30 Nov 2014, at 20:13, Gerard Meijssen wrote: Hoi, Right ? so it is thanksgiving et al.. Be thankful that it is seen, It is not Wikipedia or any of the projects...so relax.. eat some left over turkey.. Thanks, GerardM On 30 November 2014 at 09:59, K. Peachey p858sn...@gmail.com wrote: ASAP? when it's already hitting approx. five hours of down time? On 30 November 2014 at 18:14, Erik Moeller e...@wikimedia.org wrote: As noted in the server admin log [1], Phabricator is currently down due to a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP. Erik [1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log -- Erik Möller VP of Product Strategy, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Phabricator outage due to network issues, 11/29
Thanksgiving is only celebrated at this time in the US. Many of us dont celebrate it. That said downtime happens, and its a non-essential service during non working hours. Well it may be frustrating, its not the end of the world. If anyone is despretely looking for a bug to fix, they can ask on irc, im sure the regulars can think of a hundred bugs off the top of their head. I dont think Peachy was grumbling so much as looking for an accurate time frame for the solution. Re selveta's comment about distributing the db: there is standard ways of doing that (e.g. simplest would be to just use db replication, and switch master on failure), im not sure if phab is important enough to warrant that. I would probably lean to no it isnt personally. Obviously that would be an operations call. --bawolff On Nov 30, 2014 5:13 AM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Hoi, Right ? so it is thanksgiving et al.. Be thankful that it is seen, It is not Wikipedia or any of the projects...so relax.. eat some left over turkey.. Thanks, GerardM On 30 November 2014 at 09:59, K. Peachey p858sn...@gmail.com wrote: ASAP? when it's already hitting approx. five hours of down time? On 30 November 2014 at 18:14, Erik Moeller e...@wikimedia.org wrote: As noted in the server admin log [1], Phabricator is currently down due to a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP. Erik [1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log -- Erik Möller VP of Product Strategy, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Phabricator outage due to network issues, 11/29
Hoi, The argument about non-working hours is problematic. When the only thing that counts are the working hours of staff in the USA you may be right. As it is, WIkimedia Germany has staff working at other times and they are affected. Affected are the non-professionals as well.. My advise, do not go there. It is broken and it needs fixing. Thanks, GerardM On 30 November 2014 at 19:02, Brian Wolff bawo...@gmail.com wrote: Thanksgiving is only celebrated at this time in the US. Many of us dont celebrate it. That said downtime happens, and its a non-essential service during non working hours. Well it may be frustrating, its not the end of the world. If anyone is despretely looking for a bug to fix, they can ask on irc, im sure the regulars can think of a hundred bugs off the top of their head. I dont think Peachy was grumbling so much as looking for an accurate time frame for the solution. Re selveta's comment about distributing the db: there is standard ways of doing that (e.g. simplest would be to just use db replication, and switch master on failure), im not sure if phab is important enough to warrant that. I would probably lean to no it isnt personally. Obviously that would be an operations call. --bawolff On Nov 30, 2014 5:13 AM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Hoi, Right ? so it is thanksgiving et al.. Be thankful that it is seen, It is not Wikipedia or any of the projects...so relax.. eat some left over turkey.. Thanks, GerardM On 30 November 2014 at 09:59, K. Peachey p858sn...@gmail.com wrote: ASAP? when it's already hitting approx. five hours of down time? On 30 November 2014 at 18:14, Erik Moeller e...@wikimedia.org wrote: As noted in the server admin log [1], Phabricator is currently down due to a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP. Erik [1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log -- Erik Möller VP of Product Strategy, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Phabricator outage due to network issues, 11/29
On Nov 30, 2014 2:07 PM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Hoi, The argument about non-working hours is problematic. When the only thing that counts are the working hours of staff in the USA you may be right. As it is, WIkimedia Germany has staff working at other times and they are affected. Affected are the non-professionals as well.. My advise, do not go there. It is broken and it needs fixing. Thanks, GerardM I just simply meant that it didnt happen in the middle of the normal day of work for the people responsible for fixing it (not neccesarily the people affected), so there is probably going to be less relavent people around (given its a weekend and a US holiday, although i imagine there are still people on-call) hence we should be gracious with our expectations. I in no way meant to suggest it shouldnt be fixed or that it shouldnt be fixed quickly (in fact i was trying to argue against the go eat turkey setiment) I didnt think WM-DE had people working on satutdays... but volunteers certainly do work on saturdays and were affected. --bawolff ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
