I know of at least one good A/B testing service that uses JavaScript at the Front End. So that would work and it wouldn't not touch the PIO+UR stack. But your answer made me think of something else: Is there a way to score two models with PIO? Maybe Google Analytics is still preferable even for that task...
Thanks Gustavo On Mon, Sep 26, 2016 at 7:21 PM, Pat Ferrel <[email protected]> wrote: > A/B testing it the typical way to compare the performance of 2 or more > versions of an app. You should make sure that the versions only vary in 1 way > to isolate what you are comparing. So channel some traffic to a page “without > recommendations” and some amount of traffic to a page “with recommendations” > and measure your conversion KPI for both cases. If your conversion is a sale > or purchase you may have to split all users permanently over the test period. > if you are only measuring CTR you may be able to split impressions since you > don’t have to track users as a group all the way to a purchase. > > On Sep 26, 2016, at 5:45 AM, Gustavo Frederico > <[email protected]> wrote: > > Allow me to elaborate some more: there's a second question related to > the original one. How to score a model in Production? > > Thanks > > On Fri, Sep 23, 2016 at 2:58 PM, Gustavo Frederico > <[email protected]> wrote: >> Considering PIO + the UR template, what are some of the ways to score >> the model? I can think of some A/B testing but that would imply >> relying on things outside the PIO + UR stack. Is there a way to >> measure lift or "score the model"? Cross-validation with separate >> testing set? >> >> Thanks >> >> Gustavo >
