Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
On 02/20/2013 12:04 PM, Luke Welling WMF wrote: I am strongly of the opinion that within broad ranges deployment frequency does not matter. It really does not matter if you deploy twice an hour or every second day. What teams deploy every second day? But, having the machinery to make it so that you could deploy twice an hour if you wanted to is all kinds of valuable. Putting time into building: * Continuous integration with build-on-commit * Tests with good coverage * A staging environment that reflects production * Managed configuration * Scripted deployment to a large number of machines pays dividends in uptime, ops sanity and developer productivity even if you only use that machinery every few days. We have some of that, but heading further down that road would be a good thing even if we chose to keep organized periodic deployments. I couldn't agree more. It's not that I feel a need of deploying something every single day, I just want the whole process to be easier and less scary, so that it doesn't disrupt everyone's work for half a day. Also, it shouldn't be a big deal to push a quick fix in between deployments. -- Juliusz ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
On 02/20/2013 05:56 PM, Arthur Richards wrote: Once we have MobileFrontend working properly (in a production-like manner) on betalabs, we'll be a big step closer to this - particularly once betalabs is continuously updating to head of master for MobileFrontend. I believe a majority of the problems we find during MobileFrontend deployments would get caught during QA/testing in this environment. That would be very helpful for E3 too. It would be a bonus if alternate branches (e.g. a feature branch could be automatically deployed to a different wiki (e.g. in labs). Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
On 02/20/2013 12:15 PM, Mark A. Hershberger wrote: I think there is a lot of ground to cover before we get more continuous deployments, but what were you thinking we needed? A simpler and faster deployment process. The part of the process when supervision is needed shouldn't, in my opinion, take more than 15 minutes (or less). Later, populating everything to all the machines should be fully automatic and not require any kind of human supervision. A report should be generated at the end. If something goes wrong, then an alert should be sent to someone responsible. I never deployed on that scale before but I did read a bit how other projects deal with this and talked to some people. Definitely, a staging environment which mirrors production would help a lot. I hope we'll get there soon. I know that Facebook has something like an internal bleeding edge version that is used by their employees. It works with the same database as the official Facebook so employees can use it just like they would use Facebook any day. This gives them an opportunity to test many things manually before they go live for millions of people without actually wasting time on boring testing, but just by using the product. I don't know the details but I think they sync the production with bleeding edge once or twice a week. -- Juliusz ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
- Original Message - From: Juliusz Gonera jgon...@wikimedia.org On 02/20/2013 12:04 PM, Luke Welling WMF wrote: I am strongly of the opinion that within broad ranges deployment frequency does not matter. It really does not matter if you deploy twice an hour or every second day. What teams deploy every second day? The ones who accidentally shipped a brown-paper-bag bug 2 days ago. ;-) Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
Sorry for digging up an old thread, but today I also started wondering if there's a way of making our deployments simpler and faster. I'm not a big fan of special highly orchestrated events when the whole team gathers and waits and then looks for regressions after deploying dozens of commits at the same time. I've been reading a bit and it's a fact that some projects do Continuous Deployment and it works for them: http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ http://www.slideshare.net/mikebrittain/mbrittain-continuous-deploymentalm3public Is there any interest at WMF in taking this path at some point? -- Juliusz On 12/26/2012 09:31 AM, Chris McMahon wrote: Hi, A number of people I know of have ideas and aspirations pertaining to a DevOps-style deployment process, a.k.a Continuous Deployment. In recent times a number of pieces of such a system have become functional: Zuul, Jenkins enhancements for tests, automated acceptance tests, etc. But looking at mediawiki.org I don't see any sort of central discussion of overall approach/design/process for DevOps/Continuous Deployment. Is it time to start such a discussion? Or is this premature? -Chris ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
I am strongly of the opinion that within broad ranges deployment frequency does not matter. It really does not matter if you deploy twice an hour or every second day. But, having the machinery to make it so that you could deploy twice an hour if you wanted to is all kinds of valuable. Putting time into building: * Continuous integration with build-on-commit * Tests with good coverage * A staging environment that reflects production * Managed configuration * Scripted deployment to a large number of machines pays dividends in uptime, ops sanity and developer productivity even if you only use that machinery every few days. We have some of that, but heading further down that road would be a good thing even if we chose to keep organized periodic deployments. Luke Welling On Wed, Feb 20, 2013 at 2:04 PM, Juliusz Gonera jgon...@wikimedia.orgwrote: Sorry for digging up an old thread, but today I also started wondering if there's a way of making our deployments simpler and faster. I'm not a big fan of special highly orchestrated events when the whole team gathers and waits and then looks for regressions after deploying dozens of commits at the same time. I've been reading a bit and it's a fact that some projects do Continuous Deployment and it works for them: http://radar.oreilly.com/2009/**03/continuous-deployment-5-**eas.htmlhttp://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html http://timothyfitz.com/2009/**02/10/continuous-deployment-** at-imvu-doing-the-impossible-**fifty-times-a-day/http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ http://www.slideshare.net/**mikebrittain/mbrittain-**continuous-** deploymentalm3publichttp://www.slideshare.net/mikebrittain/mbrittain-continuous-deploymentalm3public Is there any interest at WMF in taking this path at some point? -- Juliusz On 12/26/2012 09:31 AM, Chris McMahon wrote: Hi, A number of people I know of have ideas and aspirations pertaining to a DevOps-style deployment process, a.k.a Continuous Deployment. In recent times a number of pieces of such a system have become functional: Zuul, Jenkins enhancements for tests, automated acceptance tests, etc. But looking at mediawiki.org I don't see any sort of central discussion of overall approach/design/process for DevOps/Continuous Deployment. Is it time to start such a discussion? Or is this premature? -Chris __**_ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/**mailman/listinfo/wikitech-lhttps://lists.wikimedia.org/mailman/listinfo/wikitech-l __**_ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/**mailman/listinfo/wikitech-lhttps://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
On 02/20/2013 02:04 PM, Juliusz Gonera wrote: Sorry for digging up an old thread, but today I also started wondering if there's a way of making our deployments simpler and faster. Wikimedia is already doing deployments every other week. On top of that, a wiki that gets a lot of use and is used by the number of core committers (translatewiki.net) does deployments on a daily (or more?) basis. At this point, I don't think they get into a huddle to monitor deployments (though I could certainly be wrong). So, the software gets a lot of use that way. I think there is a lot of ground to cover before we get more continuous deployments, but what were you thinking we needed? -- http://hexmode.com/ There is no path to peace. Peace is the path. -- Mahatma Gandhi, Non-Violence in Peace and War ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
On Wed, Feb 20, 2013 at 12:04 PM, Juliusz Gonera jgon...@wikimedia.orgwrote: I'm not a big fan of special highly orchestrated events when the whole team gathers and waits and then looks for regressions after deploying dozens of commits at the same time. I don't know if this is true of other teams (I assume it's not), but this has been how the mobile team has been doing deployments. Some reasons for this are that we currently do not have automated tests, we need myriad device testing coverage, unexpected things have a tendency to go wrong only in production, the codebase is evolving very rapidly (even our weekly deployments can be rather larger and hairy), etc. We don't necessarily need all hands on deck for deployments, but it makes discovering regressions and debugging things that go sideways only in production a lot faster. Is there any interest at WMF in taking this path at some point? Once we have MobileFrontend working properly (in a production-like manner) on betalabs, we'll be a big step closer to this - particularly once betalabs is continuously updating to head of master for MobileFrontend. I believe a majority of the problems we find during MobileFrontend deployments would get caught during QA/testing in this environment. I know the QA team has been talking about coordinating automated testing for mobile, but I'm not sure what the current status is - hopefully sooner rather than later ;) Once these two pieces are in place, MobileFrontend deployments should be much less painful and require less focus from the entire team, and we'll be a stone's throw from the glorious future of continuous deployment :D -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
On Wed, Feb 20, 2013 at 2:56 PM, Arthur Richards aricha...@wikimedia.orgwrote: I know the QA team has been talking about coordinating automated testing for mobile, but I'm not sure what the current status is - hopefully sooner rather than later ;) Everything is set up. The only missing piece (quoting you): betalabs is continuously updating to head of master for MobileFrontend. When we have that it will make sense to run browser tests (iOS and Androdid) after every commit. For now they run at least twice a day: https://wmf.ci.cloudbees.com/ If you have any questions, let me know. I am in San Francisco for the next couple of weeks, if somebody wants to talk in person. Željko ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
On Wed, Feb 20, 2013 at 4:12 PM, Željko Filipin zfili...@wikimedia.orgwrote: On Wed, Feb 20, 2013 at 2:56 PM, Arthur Richards aricha...@wikimedia.org wrote: I know the QA team has been talking about coordinating automated testing for mobile, but I'm not sure what the current status is - hopefully sooner rather than later ;) Everything is set up. The only missing piece (quoting you): betalabs is continuously updating to head of master for MobileFrontend. When we have that it will make sense to run browser tests (iOS and Androdid) after every commit. Thanks Željko! This is so rad. We are very close :) -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
On 26 December 2012 17:31, Chris McMahon cmcma...@wikimedia.org wrote: Is it time to start such a discussion? Or is this premature? Everyone has to first read and understand (possibly with shuddering) https://twitter.com/DEVOPS_BORAT - d. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] DevOps/Continuous Deployment discussion?
Yes, I am seeking a Continuous Deployment solution for Limn. I would be glad to participate in a centralization effort. On Wed, Dec 26, 2012 at 12:38 PM, David Gerard dger...@gmail.com wrote: On 26 December 2012 17:31, Chris McMahon cmcma...@wikimedia.org wrote: Is it time to start such a discussion? Or is this premature? Everyone has to first read and understand (possibly with shuddering) https://twitter.com/DEVOPS_BORAT - d. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
