It was some time ago, and I believe we had to modify the dashboard in order to show the metrics split out. I don't have access to that code any more, so I can't really be 100% sure how we did it.
Sorry, Rob On Nov 6, 2012, at 1:21 PM, Ian Young <[email protected]> wrote: > Thanks Rob. I did test Vanity and couldn't figure out how to get multiple > metrics working. Playing with it again, I think I now see how the feature > works and why it didn't match what I expected to see. Maybe you can pitch in > to tell me if I'm understanding it correctly. > > In Vanity, if you define multiple metrics for an experiment, calling track!() > on any one of those metrics will mark the user as converted. But it all goes > into the same conversion count in the dashboard. So I could track > :account_created and :newsletter_signup as two different metrics, and get > conversion rates for an experiment based on how many users did *at least one* > of those things. > > A nice enough feature, but unfortunately not quite the functionality I was > hoping for - I'd like to track and view each metric separately. Anyone used > Absurdity and know if it works the Vanity way, or my hypothetical way? > > Ian > > > On Tue, Nov 6, 2012 at 12:02 PM, Robert Kaufman <[email protected]> wrote: > So you want to track multiple metrics for a given A/B test? That would be an > Absurdity https://github.com/xing/absurdity! Oh, the Vanity > http://vanity.labnotes.org/ of some people. ;-D > > Best, > Rob > > PS: I know the Vanity docs don't really describe using more than one metric, > but it has worked for me in the past and it does appear from > http://vanity.labnotes.org/api/Vanity/Experiment/AbTest.html#metrics-instance_method > that it should continue to work. > > > On Nov 2, 2012, at 1:35 PM, Ian Young <[email protected]> wrote: > >> Hey all, >> >> I'm looking into A/B testing tools for Rails. There are a number of good >> options available, but to my surprise, none of the tools I've looked at >> appear to support measuring multiple metrics for a single test. I know I'm >> not the only one to think of this, because >> http://bjk5.com/post/28269263789/lessons-learned-a-b-testing-with-gae-bingo >> describes and screenshots exactly what I had in mind. Unfortunately, that's >> a Python tool he's talking about. >> >> It's easy enough to optimize your landing page based on a single metric >> (conversions), but like Khan Academy, I want to test more nuanced questions >> - are our users more engaged with the content? Do they participate more? Can >> we increase click-through without harming long-term retention? It seems >> somewhat constraining to test a new feature but only get an answer for a >> single measurement. >> >> So, anyone know of a tool I've overlooked that might do this? Am I really >> the only one who wants this feature? Anyone have a persuasive argument in >> favor of sticking to a single metric? >> >> Ian >> >> -- >> SD Ruby mailing list >> [email protected] >> http://groups.google.com/group/sdruby > > > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > > > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby
