Oh, sorry, I was aware of the security hole and the Ops team's awesome (as usual) efforts to patch it. And I know this particular outage was due to the security hole, and I don't think anyone could reasonably expect it to have gone much better. But, in general, things like the updates to NFS have caused problems for people who value stability over improvements. Sorry if I brought this up in the middle of a different thread though.
As always, team Analytics appreciates all the great work that's done to keep Labs up. A lot of our tools run in Labs and that's great for many reasons. On Fri, Jan 30, 2015 at 4:20 PM, John <[email protected]> wrote: > This wasnt a case of improvements. This was an emergency patching of a > security hole. All WMF servers had to be updated and rebooted, the only > noticeable reboots where in labs where there isnt redundancy/standby > services that can be swapped into place to prevent down time. > > This is not to say that our process went without issues. I will be > drafting a postmortem from the user's perspective with several ideas and > suggestions this weekend. > > On Fri, Jan 30, 2015 at 4:11 PM, Dan Andreescu <[email protected]> > wrote: > >> FYI: some of us in Analytics are talking about a "graduate labs" kind of >> environment. Which would be more stable and treated a bit less >> experimentally. It's not something we're going to push for yet, but I was >> wondering how people here felt about it. It feels like our current labs >> environment is improving at an amazing rate, but sometimes we might want to >> prioritize stability over improvement. >> >> On Fri, Jan 30, 2015 at 4:05 PM, Harry Burt <[email protected]> wrote: >> >>> Okay, gonna do something about my tools, they fall over every time this >>> happens. >>> >>> Is it just a case of echo "webservice" > ".bigbrotherrc" ? Or does the >>> daemon have to be enabled? >>> >>> Harry >>> >>> -- >>> Harry Burt (User:Jarry1250) >>> >>> >>> _______________________________________________ >>> Labs-l mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/labs-l >>> >>> >> >> _______________________________________________ >> Labs-l mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/labs-l >> >> > > _______________________________________________ > Labs-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/labs-l > >
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
