In a movement like this with a lot of very active, very leveraged community activity, it seems to me that we should *always* be trying to make things that are infrastructure instead of closed products.
cc Halfak - one of the few talks I managed to attend at Wikimania was his talk on "Research as Infrastructure", which I thought made the case very well. *Edward Saperia* Conference Director Wikimania London <http://www.wikimanialondon.org> email <[email protected]> • facebook <http://www.facebook.com/edsaperia> • twitter <http://www.twitter.com/edsaperia> • 07796955572 133-135 Bethnal Green Road, E2 7DG On 3 September 2014 00:17, Jonathan Morgan <[email protected]> wrote: > I agree with you, Ed. Although I don't think that it's realistic to expect > a product teamlike EE/Growth to create these open research tools. Their > primary output is always going to be the shiny products, not the > slightly-less-shiny infrastructure. Now *Analytics, *on the other hand.. > (*coughs* and looks pointedly at Ironholds...). > > Also, the next round of IEGs opened yesterday > <https://meta.wikimedia.org/wiki/Grants:IEG>. There's probably a fundable > project in what you describe, given a team with the right skill sets. I'd > be happy to provide feedback on a proposal. > > Cheers, > Jonathan > > > > > On Mon, Sep 1, 2014 at 8:16 AM, Edward Saperia <[email protected]> > wrote: > >> Sure, I understand how research is done. >> >> However, you could feasibly create components that allow for a certain >> types of experiments and open up the analysis side to the community. I >> think this could be a lot more successful than you'd expect - the community >> has many smart people, and together we could decide and promote best >> practice across projects/experiments. They'd also be able to drive >> suggestions for what new components to implement to expand the experiment >> space, and more generally grow interest in the work of the EE team. >> >> I understand that what you're doing now is quick and dirty and just >> trying to get something up and working, but I hope that longer term you >> have in mind the capability of the community to help you in this kind of >> endeavour. We're all keen to grow participation, and giving us tools to >> experiment ourselves will ultimately be more effective than anything you >> can do centrally. >> >> *Edward Saperia* >> Conference Director Wikimania London <http://www.wikimanialondon.org> >> email <[email protected]> • facebook >> <http://www.facebook.com/edsaperia> • twitter >> <http://www.twitter.com/edsaperia> • 07796955572 >> 133-135 Bethnal Green Road, E2 7DG >> >> >> On 26 August 2014 20:03, Oliver Keyes <[email protected]> wrote: >> >>> Neither; the tools we have for running experiments are largely >>> hand-build on an ad hoc basis. For data collection we have tools like >>> eventlogging, although they require developer energy to integrate with >>> [potential area of experimentation]. But for actually analysing the results >>> it looks very different. >>> >>> Let's use a couple of concrete examples: suppose we wanted to look at >>> whether there was a statistically significant variation in whether or not >>> people edited if we included a contributor tagline, versus didn't. We'd >>> need to take the same set of pages, ideally, and run a controlled study >>> around an A/B test. >>> >>> So first we'd display one version of the site for 50% of the population >>> and another for the other 50% (realistically we'd probably use smaller sets >>> and give the vast majority of editors the default experience, but it's a >>> hypothetical, so let's run with it). That would require developer energy. >>> Then we'd set up some kind of logging to pipe back edit attempts and view >>> attempts by [control sample/not control sample]. Also developer energy, >>> although much less. *Then*, crucially, we'd have to actually do the >>> analysis, which is not something that can be robustly generalised. >>> >>> In this example we'd be looking for significance, so we'd be looking at >>> using some kind of statistical hypothesis test. Those vary depending on >>> what probability distributions the underlying population follows. So we >>> need to work out what probability distribution is most appropriate, and >>> then apply the test most appropriate to that distribution. And that's not >>> something that can be automated through software. As a result, we get the >>> data and then work out how to test for significance. >>> >>> The alternate hypothesis would be something observational; you make the >>> change and then compare the behaviour of people while the change is live to >>> their behaviour before and after. This cuts out most of the developer cost >>> but doesn't do anything for the research support or the ad-hoc code and >>> tools that need to come with it. >>> >>> >>> On 26 August 2014 10:52, Edward Saperia <[email protected]> wrote: >>> >>>> You mean, you don't have them yourselves, or you can't expose them? >>>> >>>> *Edward Saperia* >>>> Conference Director Wikimania London <http://www.wikimanialondon.org> >>>> email <[email protected]> • facebook >>>> <http://www.facebook.com/edsaperia> • twitter >>>> <http://www.twitter.com/edsaperia> • 07796955572 >>>> 133-135 Bethnal Green Road, E2 7DG >>>> >>>> >>>> On 26 August 2014 15:46, Oliver Keyes <[email protected]> wrote: >>>> >>>>> Except we don't have those tools. There are a lot of domains in the >>>>> ecosystem where this kind of experimentation and targeting on a per-wiki >>>>> or >>>>> per-project basis, but we have a big gap around functionality and >>>>> expertise >>>>> to let us scientifically test the efficacy of their various >>>>> implementations. >>>>> >>>>> >>>>> On 26 August 2014 10:34, Edward Saperia <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>>> There's no point in polling existing community members about >>>>>>> functionality they will not see. >>>>>>> >>>>>> >>>>>> While I am a great supporter of your team's work, I'd just like to >>>>>> comment on the above; >>>>>> >>>>>> Wiser community members are aware that they are part of a powerful >>>>>> ecosystem, and that taming this ecosystem is a far more leveraged pursuit >>>>>> than doing the work yourself. Creating additional endpoints for >>>>>> onboarding >>>>>> processes that you're exposing to new users should be something that all >>>>>> projects are excited to take part in, so hopefully you'd want to poll the >>>>>> community for the valuable "Yes, and..." responses you'll get. >>>>>> >>>>>> If you find you don't get responses like this, perhaps you might want >>>>>> to consider re-framing your new functionality as open infrastructure that >>>>>> the rest of the community is invited to build on, for example maybe >>>>>> wikiprojects themselves could specify the suggestions that are shown to >>>>>> new >>>>>> editors who edit in their subject areas? >>>>>> >>>>>> Given appropriate tools to track effectiveness, this could create a >>>>>> huge, open environment for experimentation that could find interesting >>>>>> solutions faster than any engineering department ever could on their own. >>>>>> >>>>>> *Edward Saperia* >>>>>> Conference Director Wikimania London >>>>>> <http://www.wikimanialondon.org/> >>>>>> email <[email protected]> • facebook >>>>>> <http://www.facebook.com/edsaperia> • twitter >>>>>> <http://www.twitter.com/edsaperia> • 07796955572 >>>>>> 133-135 Bethnal Green Road, E2 7DG >>>>>> >>>>>> _______________________________________________ >>>>>> EE mailing list >>>>>> [email protected] >>>>>> https://lists.wikimedia.org/mailman/listinfo/ee >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Oliver Keyes >>>>> Research Analyst >>>>> Wikimedia Foundation >>>>> >>>>> _______________________________________________ >>>>> EE mailing list >>>>> [email protected] >>>>> https://lists.wikimedia.org/mailman/listinfo/ee >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> EE mailing list >>>> [email protected] >>>> https://lists.wikimedia.org/mailman/listinfo/ee >>>> >>>> >>> >>> >>> -- >>> Oliver Keyes >>> Research Analyst >>> Wikimedia Foundation >>> >>> _______________________________________________ >>> EE mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/ee >>> >>> >> >> _______________________________________________ >> EE mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/ee >> >> > > > -- > Jonathan T. Morgan > Learning Strategist > Wikimedia Foundation > User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> > [email protected] > > > _______________________________________________ > EE mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/ee > >
_______________________________________________ EE mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/ee
