Sure: point me to something I mentioned that's a closed product and isn't a prerequisite for an open one? :p
On Wednesday, 3 September 2014, Edward Saperia <[email protected]> wrote: > In a movement like this with a lot of very active, very leveraged > community activity, it seems to me that we should *always* be trying to > make things that are infrastructure instead of closed products. > > cc Halfak - one of the few talks I managed to attend at Wikimania was his > talk on "Research as Infrastructure", which I thought made the case very > well. > > *Edward Saperia* > Conference Director Wikimania London <http://www.wikimanialondon.org> > email <javascript:_e(%7B%7D,'cvml','[email protected]');> • facebook > <http://www.facebook.com/edsaperia> • twitter > <http://www.twitter.com/edsaperia> • 07796955572 > 133-135 Bethnal Green Road, E2 7DG > > > On 3 September 2014 00:17, Jonathan Morgan <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> I agree with you, Ed. Although I don't think that it's realistic to >> expect a product teamlike EE/Growth to create these open research tools. >> Their primary output is always going to be the shiny products, not the >> slightly-less-shiny infrastructure. Now *Analytics, *on the other hand.. >> (*coughs* and looks pointedly at Ironholds...). >> >> Also, the next round of IEGs opened yesterday >> <https://meta.wikimedia.org/wiki/Grants:IEG>. There's probably a >> fundable project in what you describe, given a team with the right skill >> sets. I'd be happy to provide feedback on a proposal. >> >> Cheers, >> Jonathan >> >> >> >> >> On Mon, Sep 1, 2014 at 8:16 AM, Edward Saperia <[email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >> >>> Sure, I understand how research is done. >>> >>> However, you could feasibly create components that allow for a certain >>> types of experiments and open up the analysis side to the community. I >>> think this could be a lot more successful than you'd expect - the community >>> has many smart people, and together we could decide and promote best >>> practice across projects/experiments. They'd also be able to drive >>> suggestions for what new components to implement to expand the experiment >>> space, and more generally grow interest in the work of the EE team. >>> >>> I understand that what you're doing now is quick and dirty and just >>> trying to get something up and working, but I hope that longer term you >>> have in mind the capability of the community to help you in this kind of >>> endeavour. We're all keen to grow participation, and giving us tools to >>> experiment ourselves will ultimately be more effective than anything you >>> can do centrally. >>> >>> *Edward Saperia* >>> Conference Director Wikimania London <http://www.wikimanialondon.org> >>> email <javascript:_e(%7B%7D,'cvml','[email protected]');> • >>> facebook <http://www.facebook.com/edsaperia> • twitter >>> <http://www.twitter.com/edsaperia> • 07796955572 >>> 133-135 Bethnal Green Road, E2 7DG >>> >>> >>> On 26 August 2014 20:03, Oliver Keyes <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> >>>> Neither; the tools we have for running experiments are largely >>>> hand-build on an ad hoc basis. For data collection we have tools like >>>> eventlogging, although they require developer energy to integrate with >>>> [potential area of experimentation]. But for actually analysing the results >>>> it looks very different. >>>> >>>> Let's use a couple of concrete examples: suppose we wanted to look at >>>> whether there was a statistically significant variation in whether or not >>>> people edited if we included a contributor tagline, versus didn't. We'd >>>> need to take the same set of pages, ideally, and run a controlled study >>>> around an A/B test. >>>> >>>> So first we'd display one version of the site for 50% of the population >>>> and another for the other 50% (realistically we'd probably use smaller sets >>>> and give the vast majority of editors the default experience, but it's a >>>> hypothetical, so let's run with it). That would require developer energy. >>>> Then we'd set up some kind of logging to pipe back edit attempts and view >>>> attempts by [control sample/not control sample]. Also developer energy, >>>> although much less. *Then*, crucially, we'd have to actually do the >>>> analysis, which is not something that can be robustly generalised. >>>> >>>> In this example we'd be looking for significance, so we'd be looking at >>>> using some kind of statistical hypothesis test. Those vary depending on >>>> what probability distributions the underlying population follows. So we >>>> need to work out what probability distribution is most appropriate, and >>>> then apply the test most appropriate to that distribution. And that's not >>>> something that can be automated through software. As a result, we get the >>>> data and then work out how to test for significance. >>>> >>>> The alternate hypothesis would be something observational; you make the >>>> change and then compare the behaviour of people while the change is live to >>>> their behaviour before and after. This cuts out most of the developer cost >>>> but doesn't do anything for the research support or the ad-hoc code and >>>> tools that need to come with it. >>>> >>>> >>>> On 26 August 2014 10:52, Edward Saperia <[email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >>>>> You mean, you don't have them yourselves, or you can't expose them? >>>>> >>>>> *Edward Saperia* >>>>> Conference Director Wikimania London <http://www.wikimanialondon.org> >>>>> email <javascript:_e(%7B%7D,'cvml','[email protected]');> • >>>>> facebook <http://www.facebook.com/edsaperia> • twitter >>>>> <http://www.twitter.com/edsaperia> • 07796955572 >>>>> 133-135 Bethnal Green Road, E2 7DG >>>>> >>>>> >>>>> On 26 August 2014 15:46, Oliver Keyes <[email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>> >>>>>> Except we don't have those tools. There are a lot of domains in the >>>>>> ecosystem where this kind of experimentation and targeting on a per-wiki >>>>>> or >>>>>> per-project basis, but we have a big gap around functionality and >>>>>> expertise >>>>>> to let us scientifically test the efficacy of their various >>>>>> implementations. >>>>>> >>>>>> >>>>>> On 26 August 2014 10:34, Edward Saperia <[email protected] >>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>>> There's no point in polling existing community members about >>>>>>>> functionality they will not see. >>>>>>>> >>>>>>> >>>>>>> While I am a great supporter of your team's work, I'd just like to >>>>>>> comment on the above; >>>>>>> >>>>>>> Wiser community members are aware that they are part of a powerful >>>>>>> ecosystem, and that taming this ecosystem is a far more leveraged >>>>>>> pursuit >>>>>>> than doing the work yourself. Creating additional endpoints for >>>>>>> onboarding >>>>>>> processes that you're exposing to new users should be something that all >>>>>>> projects are excited to take part in, so hopefully you'd want to poll >>>>>>> the >>>>>>> community for the valuable "Yes, and..." responses you'll get. >>>>>>> >>>>>>> If you find you don't get responses like this, perhaps you might >>>>>>> want to consider re-framing your new functionality as open >>>>>>> infrastructure >>>>>>> that the rest of the community is invited to build on, for example maybe >>>>>>> wikiprojects themselves could specify the suggestions that are shown to >>>>>>> new >>>>>>> editors who edit in their subject areas? >>>>>>> >>>>>>> Given appropriate tools to track effectiveness, this could create a >>>>>>> huge, open environment for experimentation that could find interesting >>>>>>> solutions faster than any engineering department ever could on their >>>>>>> own. >>>>>>> >>>>>>> *Edward Saperia* >>>>>>> Conference Director Wikimania London >>>>>>> <http://www.wikimanialondon.org/> >>>>>>> email <javascript:_e(%7B%7D,'cvml','[email protected]');> • >>>>>>> facebook <http://www.facebook.com/edsaperia> • twitter >>>>>>> <http://www.twitter.com/edsaperia> • 07796955572 >>>>>>> 133-135 Bethnal Green Road, E2 7DG >>>>>>> >>>>>>> _______________________________________________ >>>>>>> EE mailing list >>>>>>> [email protected] >>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');> >>>>>>> https://lists.wikimedia.org/mailman/listinfo/ee >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Oliver Keyes >>>>>> Research Analyst >>>>>> Wikimedia Foundation >>>>>> >>>>>> _______________________________________________ >>>>>> EE mailing list >>>>>> [email protected] >>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');> >>>>>> https://lists.wikimedia.org/mailman/listinfo/ee >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> EE mailing list >>>>> [email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');> >>>>> https://lists.wikimedia.org/mailman/listinfo/ee >>>>> >>>>> >>>> >>>> >>>> -- >>>> Oliver Keyes >>>> Research Analyst >>>> Wikimedia Foundation >>>> >>>> _______________________________________________ >>>> EE mailing list >>>> [email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');> >>>> https://lists.wikimedia.org/mailman/listinfo/ee >>>> >>>> >>> >>> _______________________________________________ >>> EE mailing list >>> [email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');> >>> https://lists.wikimedia.org/mailman/listinfo/ee >>> >>> >> >> >> -- >> Jonathan T. Morgan >> Learning Strategist >> Wikimedia Foundation >> User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> >> [email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');> >> >> >> _______________________________________________ >> EE mailing list >> [email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');> >> https://lists.wikimedia.org/mailman/listinfo/ee >> >> > -- Sent from a portable device of Lovecraftian complexity.
_______________________________________________ EE mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/ee
