Do we have a timeline on when we want to do those quick surveys? Cheers,
Deb On Fri, Jan 22, 2016 at 2:08 PM, Trey Jones <[email protected]> wrote: > Cool! I guess I had in my mind from previous and incomplete conversation > that the goal was a specific cutoff. Thanks for the clarification. > > Trey Jones > Software Engineer, Discovery > Wikimedia Foundation > > On Fri, Jan 22, 2016 at 4:05 PM, Mikhail Popov <[email protected]> > wrote: > >> That's actually our goal with quick surveys :P We want to ask users for >> their satisfaction with our search and then build a predictive model with >> satisfaction as the response variable and dwell time + other data as the >> predictor variables. >> >> Right now we're stuck at the "get training data" step. Once that's >> resolved, we can do precisely what you described :D Then we'll have a daily >> estimate of user satisfaction (unobservable without direct user feedback) >> using data we can observe (browsing behavior). >> >> Thanks, >> Mikhail >> >> On Fri, Jan 22, 2016 at 11:19 AM, Trey Jones <[email protected]> >> wrote: >> >>> Yesterday in the quarterly review Dan mentioned that our current user >>> satisfaction metric uses the somewhat arbitrary 10s dwell time cutoff for a >>> successful search, and that we want to use a survey to correlate >>> qualitative and quantitative values to pin down a better cutoff for our >>> users. I don't remember whether Dan mentioned it, or I was just rehashing >>> the notion on my own, but it may be difficult to pin down a specific cutoff. >>> >>> A wild thought appears! Why do we have to pin down a specific cut off? >>> Why can't we have a probabilistic user satisfaction metric? (Other then >>> complexity and computational speed, which may be relevant.) >>> >>> We have the ability to gather so much data that we could easily compute >>> something like this: 20% of users are satisfied when dwell time is <5s, 35% >>> for 5-10s, 75% for 10-60s, 98% for 1m-5m, 85% for 5m-20m, and 80% for >20m. >>> >>> Determining the cutoffs might be tricky, and computation is more complex >>> than counting, but not ridiculously complicated, and potentially much more >>> accurate for large samples. Presenting the results is still easy: "54.7% of >>> our users are happy with their search results based on our dwell-time >>> model". >>> >>> I tried to do a quick search for papers on this topic, but I didn't find >>> anything. I'm not familiar with the literature, so that may not mean much. >>> >>> Okay, back to the TextCat mines.... >>> >>> —Trey >>> >>> Trey Jones >>> Software Engineer, Discovery >>> Wikimedia Foundation >>> >>> _______________________________________________ >>> discovery mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/discovery >>> >>> >> >> >> -- >> *Mikhail Popov* // Data Analyst, Discovery >> <https://www.mediawiki.org/wiki/Wikimedia_Discovery> >> https://wikimediafoundation.org/ >> >> *Imagine a world in which every single human being can freely share in >> the **sum of all knowledge. That's our commitment.* Donate >> <https://donate.wikimedia.org/>. >> >> _______________________________________________ >> discovery mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/discovery >> >> > > _______________________________________________ > discovery mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/discovery > > -- -- Deb Tankersley Product Manager, Discovery Wikimedia Foundation
_______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
