CC'ing Greg to help us think how to approach percentage based releases and what would be in our way to do them.
--tomasz On Tue, Jan 27, 2015 at 1:18 PM, Jon Katz <[email protected]> wrote: > Jon, > Are there any technical reasons not to explore the % rollout model for > collections instead of beta? > -J > > > On Tue, Jan 27, 2015 at 11:20 AM, Maryana Pinchuk <[email protected]> > wrote: >> >> Yeah, I think there are a lot of problems with the opt-in beta model. I >> much prefer releasing new features to a small % of users, logging >> events/usage, and if we suspect something has the potential to be >> disruptive/offputting, letting them know the feature they're seeing is beta >> and letting them turn it off. >> >> That said, beta is still useful for sandboxing new features and in-person >> user testing, so I don't think we should kill it altogether. I just think we >> need to supplement it with a graduated release model – which we're already >> doing with stuff like WikiGrok :) >> >> On Tue, Jan 27, 2015 at 11:10 AM, Sam Smith <[email protected]> >> wrote: >>>> >>>> Have we thought about automatically opting people into beta mode e.g. >>>> a sample of our users in a certain geographic region / certain zero >>>> enabled area/ all users in a certain bucket based on their user id ? >>> >>> >>> I like this idea. In fact, I'm for it, provided that we make it clear to >>> the user that they've been entered into an experiment and they're seeing >>> non-standard UI. >>> >>>> How many users could beta actually handle? >>> >>> >>> Not sure. But, interestingly, we can find out by bucketing users and >>> slowly assigning them the beta variant. >>> >>>> Is this technically possible? >>> >>> >>> Yes. If we're generating and storing tokens on the client, which we do >>> for anonymous users in other experiments, then we can enter anonymous users >>> into the experiment at the cost of a little control over how tokens are >>> stored. >>> >>>> If someone was bucketed into beta would they be able to opt out into >>>> stable again under any of the above situations? >>> >>> >>> See my first inline response. We must make it clear to the user that >>> they're seeing a variant of an experiment… and we must make it simple to opt >>> out of the experiment. >>> >>> Also, all instrumentation for beta features will need to be augmented >>> with a is_beta_opt_in flag. >>> >>> –Sam >>> >>> On Tue, Jan 27, 2015 at 10:18 AM, Jon Robson <[email protected]> >>> wrote: >>>> >>>> One of the frustrations I have heard so far is that the audience there >>>> is too small to get meaningful data around various experiments. >>>> Currently people have to opt in by going to >>>> https://en.m.wikipedia.org/wiki/Special:MobileOptions which is hidden >>>> away in the mobile interface. They can do this whilst anonymous or >>>> logged in. >>>> >>>> Have we thought about automatically opting people into beta mode e.g. >>>> a sample of our users in a certain geographic region / certain zero >>>> enabled area/ all users in a certain bucket based on their user id ? >>>> >>>> How many users could beta actually handle? >>>> Is this technically possible? >>>> If someone was bucketed into beta would they be able to opt out into >>>> stable again under any of the above situations? >>>> >>>> Jon >>>> >>>> _______________________________________________ >>>> Mobile-l mailing list >>>> [email protected] >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >>> >>> >>> _______________________________________________ >>> Mobile-l mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >> >> >> >> -- >> Maryana Pinchuk >> Product Manager, Wikimedia Foundation >> wikimediafoundation.org >> >> _______________________________________________ >> Mobile-l mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> > > > _______________________________________________ > Mobile-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/mobile-l > _______________________________________________ Mobile-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mobile-l
