Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
Erik Moeller wrote: What would be the disadvantage of having a single I'd like the latest and greatest changes once they come in preference for our users? The main disadvantage I see is that we'd need to temporarily retain two codepaths for significant user-facing changes, potentially increasing code complexity a fair bit, but perhaps reducing post-launch cost in return. It would increase code complexity a lot, and it would make debugging more difficult. Don't forget that the desktop execution environment (front back) is much more heterogenous -- you have gadgets, substantial interface customization via the MediaWiki NS, multiple skins, legacy interfaces, and a much wider set of extensions, all of which conspire to make bugs much harder to reproduce. I'm inclined to think that we already get most of the benefits this approach purports to confer from the general MediaWiki release cycle. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
On 05/07/2013 08:47 AM, Andre Klapper wrote:
On Mon, 2013-05-06 at 12:52 -0700, Greg Grossmeier wrote:
Also, the majority of bugs that
are in the Highest/Immediate priority level (from my gut assessment, I
don't have the data here) are found after a deploy to non-WP projects.
I agree with that impression: We don't get many (manually found)
highest/immediate prio bug reports after the first deployment phase,
most of them after phase 2, and a few after phase 3 (e.g. when we failed
to understand the explosive force of an issue).
Backing that impression up with Bugzilla data:
tl;dr: That's hard.
Long version:
I tried a Bugzilla query for tickets created in the last four months,
that at some point in their lifetime had Priority = {Highest |
Immediate}, restricted it to the products {MediaWiki, MediaWiki
extensions, Wikimedia}, made buglist.cgi display the Opened column
(via Change columns at the bottom); dropped the last 9 characters of
the Opened column (to get rid of the time and only have the date,
though that's UTC so does not perfectly fit our deployment *time*),
imported the resulting CSV into OOCalc, cumulated a bit, and summed up
all those tickets that got filed in a certain deployment phase at
*some* point became highest/immediate. See attachment.
The results don't back up my impression.
One potential reason: Development teams file tickets *at some point* and
don't see priority immediately, and when tickets get triaged they get
higher priority at some point later on. Maybe results would look
different if I the query excluded reporters that are employees?
Don't want to spend too much time trying though.
andre
Andre, thanks for trying to run the query on this. Yeah, we split,
rename, reprioritize, and otherwise mess with our Bugzilla tickets so
often that it'd probably be a huge travail to thoroughly research the
question I asked, and that would delay this decision beyond this week.
So I'm fine with going on people's rough estimates and impressions
instead of going the ALL DATA ALL THE TIME route.
--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
quote name=Sumana Harihareswara date=2013-05-07 time=10:20:05 -0400 Backing that impression up with Bugzilla data: tl;dr: That's hard. snip The results don't back up my impression. One potential reason: Development teams file tickets *at some point* and don't see priority immediately, and when tickets get triaged they get higher priority at some point later on. Maybe results would look different if I the query excluded reporters that are employees? Don't want to spend too much time trying though. Andre, thanks for trying to run the query on this. Yeah, we split, rename, reprioritize, and otherwise mess with our Bugzilla tickets so often that it'd probably be a huge travail to thoroughly research the question I asked, and that would delay this decision beyond this week. So I'm fine with going on people's rough estimates and impressions instead of going the ALL DATA ALL THE TIME route. Yeah, there isn't any pattern in that graph as is and it would need some more intense work to get right. Thanks for trying, Andre! Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
On Mon, May 6, 2013 at 7:43 PM, Erik Moeller e...@wikimedia.org wrote: We may want to consider at least putting some such scaffolding for beta-prod desktop modes into place before shifting to weekly deployments, although if that holds up this change significantly, I'd be in favor of making the shift first and then iterating. Right now we have lots of individual experimental prefs, some dark launch URL parameters (useNew=1 for the account creation forms etc.), some changes that are announced widely but then rolled out immediately (section edit link change), etc. What would be the disadvantage of having a single I'd like the latest and greatest changes once they come in preference for our users? The main disadvantage I see is that we'd need to temporarily retain two codepaths for significant user-facing changes, potentially increasing code complexity a fair bit, but perhaps reducing post-launch cost in return. And we'd need to consider more carefully if/when to make the beta/prod switch -- not necessarily a bad thing. ;-) Have there been any negative experiences with this model on the mobile sites? For the most part, this model has been awesome for mobile. It allows us to iterate on feature ideas quickly, try a lot of different things, and focus on the things that really work. Getting a partially-polished feature in front of a group of users who expect to have things not always work 100% is incredibly valuable - and takes off the pressure of having to get something totally right. If it turns out the feature idea was a bust, it's generally no big deal for us (the mobile team or the users) to either can it or tweak it to make it better. Having an 'alpha' that is essentially a developer sandobx (little to no productization happens for our alpha) and some of the best mobile features have grown from here. We've had occasional issues with beta or alpha features bleeding into a mode they weren't supposed to - though this is generally quick to fix. Ultimately, the issues we've had with the approach have been minor. If this were something adopted by Mediawiki more generally, I would want to see it carefully built into core. Ori brings up a good point about increased code complexity, which is a guarantee with this kind of approach but could certainly be mitigated if the plumbing were well architected. I think thoughtful development of something like this into core would be awesome and would allow us to collectively build better, more useful features, faster. -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
On Mon, May 6, 2013 at 7:43 PM, Erik Moeller e...@wikimedia.org wrote: On Mon, May 6, 2013 at 7:20 PM, MZMcBride z...@mzmcbride.com wrote: The reason I ask about a distinction is that there have been a lot of changes to Wikimedia wikis lately and likely more to come, as the Wikimedia Foundation has gotten larger and has more dedicated tech resources. Overall, this is great. But big new features come with big changes, and these changes sometimes need a bit of breathing room. I've read a lot of pushback lately against rapid changes (usurping usernames, getting rid of the new message indicator, etc.). A lot of this seems mostly outside the scope how often to deploy (and I don't want to shift the focus of this thread), but it gets confusing (to me, at least) to make a distinction between new code/features on Wikimedia wikis and how often to branch MediaWiki core/extensions. A lot of this could potentially be addressed in a consistent manner across wikis if we applied the alpha-beta-prod (or just beta-prod for starters) channel model that's used on the Wikimedia mobile sites. Then features (whether in core or extensions) could be flagged for alpha or beta readiness, and users would only get them if they've decided to opt into either of those channels. We could still flip the switch from beta-prod, but that decision could be decoupled from the weekly deployment cycle. This would likely be done for features changes which have significant user-facing impact, and where segregation into on and off modes is possible (not always the case). I think this is awesome for features ... but if we're putting work into this, I would love even more to have a clustered a+b production environment, such that 10% of folks are put on the new release (cluster a) and then it gets pushed over to cluster b. Then we can also test performance in a real world environment, and breakages only happen for 10% (PS the 10% number was pulled out of thin air). The opt-in beta is much too limited, as well as being inapplicable to the vast majority of our traffic (logged in users are such a small percentage) to make proper comparisons. You could also see the impact of features on usage for average users. We may want to consider at least putting some such scaffolding for beta-prod desktop modes into place before shifting to weekly deployments, although if that holds up this change significantly, I'd be in favor of making the shift first and then iterating. Right now we have lots of individual experimental prefs, some dark launch URL parameters (useNew=1 for the account creation forms etc.), some changes that are announced widely but then rolled out immediately (section edit link change), etc. What would be the disadvantage of having a single I'd like the latest and greatest changes once they come in preference for our users? The main disadvantage I see is that we'd need to temporarily retain two codepaths for significant user-facing changes, potentially increasing code complexity a fair bit, but perhaps reducing post-launch cost in return. And we'd need to consider more carefully if/when to make the beta/prod switch -- not necessarily a bad thing. ;-) Have there been any negative experiences with this model on the mobile sites? Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
quote name=Leslie Carr date=2013-05-07 time=11:43:47 -0700 I think this is awesome for features ... but if we're putting work into this, I would love even more to have a clustered a+b production environment, such that 10% of folks are put on the new release (cluster a) and then it gets pushed over to cluster b. Then we can also test performance in a real world environment, and breakages only happen for 10% (PS the 10% number was pulled out of thin air). I really like this idea, I think. So bringing various concurrent (email) threads together; long term we could have something that looks like this (roughly): 1) Change proposed - Jenkins runs tests on a throw away labs instance 2) Change merged to master - Jenkins/etc runs tests on betalabs 3) New wmfXX released to 10% of cluster - 10% being something like: test, test2, mediawiki.org, and some of the non-'pedia project sites - Our users do the testing ;-) 4) That wmfXX goes to the rest of the cluster - Hopefully all is good. Is that kind of what you had in mind, Leslie? Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
On 05/07/2013 03:02 PM, Greg Grossmeier wrote: 3) New wmfXX released to 10% of cluster - 10% being something like: test, test2, mediawiki.org, and some of the non-'pedia project sites - Our users do the testing ;-) I was interpreting Leslie as saying 10% cross-cut throughout. I.E. 10% of German Wiktionary, 10% of Japanese Wikinews, 10% of English Wikipedia, 10% of everything. That would mean we would really get wide testing before it rolled out to the 90%. But I don't know which she meant. Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
On 05/07/2013 04:46 PM, Matthew Flaschen wrote: I was interpreting Leslie as saying 10% cross-cut throughout. I.E. 10% of German Wiktionary, 10% of Japanese Wikinews, 10% of English Wikipedia, 10% of everything. That would mean we would really get wide testing before it rolled out to the 90%. For this to work for user-facing changes, though, there would have to be a clear indicator of which you're on (maybe even an automatic bug report tool), or reports would be confusing. Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
quote name=Matthew Flaschen date=2013-05-07 time=16:47:32 -0400 On 05/07/2013 04:46 PM, Matthew Flaschen wrote: I was interpreting Leslie as saying 10% cross-cut throughout. I.E. 10% of German Wiktionary, 10% of Japanese Wikinews, 10% of English Wikipedia, 10% of everything. That would mean we would really get wide testing before it rolled out to the 90%. For this to work for user-facing changes, though, there would have to be a clear indicator of which you're on (maybe even an automatic bug report tool), or reports would be confusing. Yeah, that was my first interpretation, but I wasn't sure how it'd work in our situation, so I internally modified and then wrote out the steps I did :-). Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
Leslie Carr wrote: I think this is awesome for features ... but if we're putting work into this, I would love even more to have a clustered a+b production environment, such that 10% of folks are put on the new release (cluster a) and then it gets pushed over to cluster b. Then we can also test performance in a real world environment, and breakages only happen for 10% (PS the 10% number was pulled out of thin air). The opt-in beta is much too limited, as well as being inapplicable to the vast majority of our traffic (logged in users are such a small percentage) to make proper comparisons. You could also see the impact of features on usage for average users. I thought the idea was to not annoy users with unpolished/beta features, unless they've opted in. This seems to be the rationale for mobile. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Proposal: Let's move to a one-week deploy cycle
Hello all, Let's move our MediaWiki deploy cycle to weekly instead of 2-week. This will also reduce the number of standing deployment windows throughout the week by having those projects/teams simply ride the MediaWiki train. == Current situation == Right now, a new version of MediaWiki is rolled out to the WMF cluster over a two-week period. You can see the general flow of how it works on this page describing the deploy schedule for the 1.22 release: https://www.mediawiki.org/wiki/MediaWiki_1.22/Roadmap#Schedule_for_the_deployments == What are the drawbacks of two-weeks? == These are mostly known by everyone so I'll just simply state the most obvious one :-) It takes up to 2 weeks for new features/bug fixes to be rolled out to the various Wikimedia wikis. == What would a one-week cycle look like? == This has been talked about a bit, including during the last In Town Week for WMF Engineering in late-February. I've coalesced on one proposal at: https://wikitech.wikimedia.org/wiki/Deployments/One_week This seems like a reasonable approach to me. Please respond here or on the talk page with comments/suggestions/etc. == Major benefit/goal with one-week cycle == Our current list of deployment windows during the week is pretty large, and it is not uncommon for a week to practically fill up with bug fix windows. If we moved to a weekly cycle then more of those bug-fixes could just roll out with the normal MediaWiki deploy. == Goal Timeline? == I would love to get us switched over to a one-week cycle by mid-June. Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
On Mon, 06 May 2013 21:04:39 +0200, Greg Grossmeier g...@wikimedia.org wrote: It takes up to 2 weeks for new features/bug fixes to be rolled out to the various Wikimedia wikis. Four weeks, actually, if your change gets merged just after a branching point. (2 weeks until next branching + 2 weeks until it's deployed everywhere.) Needless to say I like this idea. -- Matma Rex ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
Le 06/05/13 21:04, Greg Grossmeier a écrit : Let's move our MediaWiki deploy cycle to weekly instead of 2-week. As long as we tend to the hour, I am fine :-] Weekly deploy is a bit of a challenge since we will have less time to figure out a solution for any blocking bug, but overall I think it will be a nice improvement to get less changes deployed. I am also wondering how many manual tasks we have to handle when doing and after the deployment. If we identify them and write tools for it, that would make it later possible to do two deployment per week :-] -- Antoine hashar Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
Greg Grossmeier wrote: == What are the drawbacks of two-weeks? == These are mostly known by everyone so I'll just simply state the most obvious one :-) It takes up to 2 weeks for new features/bug fixes to be rolled out to the various Wikimedia wikis. This is not a bad thing, though. Two weeks is a helluva lot better than it could be (or has been). :-) These conversations get a little murky when we're discussing deployments. I feel like there's a distinction between MediaWiki core and extension general fixes and upgrades versus new features and functionality being deployed to Wikimedia wikis. I think everyone agrees that the faster we can get bug fixes in, the better. But when it comes to triaging and prioritizing features and enabling certain features by default... does such a distinction exist outside my mind? The reason I ask about a distinction is that there have been a lot of changes to Wikimedia wikis lately and likely more to come, as the Wikimedia Foundation has gotten larger and has more dedicated tech resources. Overall, this is great. But big new features come with big changes, and these changes sometimes need a bit of breathing room. I've read a lot of pushback lately against rapid changes (usurping usernames, getting rid of the new message indicator, etc.). A lot of this seems mostly outside the scope how often to deploy (and I don't want to shift the focus of this thread), but it gets confusing (to me, at least) to make a distinction between new code/features on Wikimedia wikis and how often to branch MediaWiki core/extensions. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
On Mon, May 6, 2013 at 7:20 PM, MZMcBride z...@mzmcbride.com wrote: The reason I ask about a distinction is that there have been a lot of changes to Wikimedia wikis lately and likely more to come, as the Wikimedia Foundation has gotten larger and has more dedicated tech resources. Overall, this is great. But big new features come with big changes, and these changes sometimes need a bit of breathing room. I've read a lot of pushback lately against rapid changes (usurping usernames, getting rid of the new message indicator, etc.). A lot of this seems mostly outside the scope how often to deploy (and I don't want to shift the focus of this thread), but it gets confusing (to me, at least) to make a distinction between new code/features on Wikimedia wikis and how often to branch MediaWiki core/extensions. A lot of this could potentially be addressed in a consistent manner across wikis if we applied the alpha-beta-prod (or just beta-prod for starters) channel model that's used on the Wikimedia mobile sites. Then features (whether in core or extensions) could be flagged for alpha or beta readiness, and users would only get them if they've decided to opt into either of those channels. We could still flip the switch from beta-prod, but that decision could be decoupled from the weekly deployment cycle. This would likely be done for features changes which have significant user-facing impact, and where segregation into on and off modes is possible (not always the case). We may want to consider at least putting some such scaffolding for beta-prod desktop modes into place before shifting to weekly deployments, although if that holds up this change significantly, I'd be in favor of making the shift first and then iterating. Right now we have lots of individual experimental prefs, some dark launch URL parameters (useNew=1 for the account creation forms etc.), some changes that are announced widely but then rolled out immediately (section edit link change), etc. What would be the disadvantage of having a single I'd like the latest and greatest changes once they come in preference for our users? The main disadvantage I see is that we'd need to temporarily retain two codepaths for significant user-facing changes, potentially increasing code complexity a fair bit, but perhaps reducing post-launch cost in return. And we'd need to consider more carefully if/when to make the beta/prod switch -- not necessarily a bad thing. ;-) Have there been any negative experiences with this model on the mobile sites? Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: Let's move to a one-week deploy cycle
No negative experiences at all Erik. The only real problems we've run into are people complaining they weren't aware of editing features that we pushed to mobile that users were not aware of due to not using mobile. I reckon this would be less of an issue in desktop. I would ___ love ___ to see a stable, beta, alpha model on desktop ... After reading a lot of the Echo feedback today I feel a lot of the problems, complaints about not hearing about it could have been addressed by being in a beta mode. It has also made innovation, experimentation, testing and transparency much easier than it would have been had we just pushed new features directly to large audiences. Please help make this happen. I'm happy to talk more in detail with anyone who wants to implement this. On 6 May 2013 19:43, Erik Moeller e...@wikimedia.org wrote: On Mon, May 6, 2013 at 7:20 PM, MZMcBride z...@mzmcbride.com wrote: The reason I ask about a distinction is that there have been a lot of changes to Wikimedia wikis lately and likely more to come, as the Wikimedia Foundation has gotten larger and has more dedicated tech resources. Overall, this is great. But big new features come with big changes, and these changes sometimes need a bit of breathing room. I've read a lot of pushback lately against rapid changes (usurping usernames, getting rid of the new message indicator, etc.). A lot of this seems mostly outside the scope how often to deploy (and I don't want to shift the focus of this thread), but it gets confusing (to me, at least) to make a distinction between new code/features on Wikimedia wikis and how often to branch MediaWiki core/extensions. A lot of this could potentially be addressed in a consistent manner across wikis if we applied the alpha-beta-prod (or just beta-prod for starters) channel model that's used on the Wikimedia mobile sites. Then features (whether in core or extensions) could be flagged for alpha or beta readiness, and users would only get them if they've decided to opt into either of those channels. We could still flip the switch from beta-prod, but that decision could be decoupled from the weekly deployment cycle. This would likely be done for features changes which have significant user-facing impact, and where segregation into on and off modes is possible (not always the case). We may want to consider at least putting some such scaffolding for beta-prod desktop modes into place before shifting to weekly deployments, although if that holds up this change significantly, I'd be in favor of making the shift first and then iterating. Right now we have lots of individual experimental prefs, some dark launch URL parameters (useNew=1 for the account creation forms etc.), some changes that are announced widely but then rolled out immediately (section edit link change), etc. What would be the disadvantage of having a single I'd like the latest and greatest changes once they come in preference for our users? The main disadvantage I see is that we'd need to temporarily retain two codepaths for significant user-facing changes, potentially increasing code complexity a fair bit, but perhaps reducing post-launch cost in return. And we'd need to consider more carefully if/when to make the beta/prod switch -- not necessarily a bad thing. ;-) Have there been any negative experiences with this model on the mobile sites? Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
