I would also point you at many of Mr. Underwood's blog posts, as they have helped me quite a bit :)
http://techblog.chegg.com/2012/12/12/measuring-search-relevance-with-mrr/ On Wed, Feb 24, 2016 at 11:37 AM, Doug Turnbull < dturnb...@opensourceconnections.com> wrote: > For relevance, I would also look at retention metrics. Harder to tie back > to a specific search. But what happens after the conversion? Did they > purchase the product and hate it? Or did they come back for more? Retention > metrics say a lot about the whole experience. But for many search-heavy > applications search is 90% of the user experience. Was it really relevant > if they purchased a product, but were dissatisfied? Did search make a > promise that wasn't delivered on? This is something I personally noodle > about and not something I have a canned solution for. > > There's an obsession with what I think of as "engagement metrics" or > "session metrics". Engagement metrics like CTR and are handy because > they're easy to tie to a search. Search, search, search <click>, search, > search, <click> <purchase>. > > I'm always cautious of click-thru metrics. Beware of the biases in your > clickthru metrics > > http://opensourceconnections.com/blog/2014/10/08/when-click-scoring-can-hurt-search-relevance-a-roadmap-to-better-signals-processing-in-search/ > > Another reason to be cautious is user behavioral data can require > domain-specific interpretation. A good book on recommendor can talk more > about interpreting user behavior to see if an item was relevant. For > examples Pratical Recommender Systems by Kim Falk (a Manning MEAP) spends a > great deal of time talking through gathering evidence whether a user liked > the thing they clicked on or not. For example, did the user click a movie > and go back immediately? Start watching a movie and go back in 5 minutes > indicating they hated it? Or watch a movie all the way through? > > Related to interpreting behavior -- understand the kinds of searchers out > there. Understand what sort of user experience you've built. Informational > searchers doing research will look at every item and evaluate them. For > example, a paralegal searching a legal application may need to examine > every result carefully. Navigational searchers want to hunt for one thing. > Everyday e-commerce searchers clicking on every result is probably > disastrous. However, the purchasing dept of an organization MIGHT look at > every result and that might be ok. > > Beware of search's long tail. You can gather metrics on all your searches > and where users are clicking, but search has a notorious long tail. Many of > my clients have meaningful metrics over perhaps the top 50 results before > quickly going off into obscurity of statistical insignificance per search. > This depends entirely on the type of search application you're developing. > Some kind of niche product with a handful of searches per day? Or giant > e-commerce site? > > Sometimes what's simpler is to do usability testing or to sit with an > expert user and gather relevance judgments--grades on what's relevant and > what's not. (this is what we do with Quepid). This works particularly well > for these niche, expert search subjects > > Anyway, there's still quite a bit of art to interpreting search metrics. I > would argue to keep the human and domain expert in the loop understanding > and interpreting metrics. But its a yin-and-yang. You also need to be able > to tell that supposed domain expert when they're wrong. > > Sorry for long winded email, but these topics dominate my > dreams/nightmares these days :) > > Best > -Doug > > > > > On Wed, Feb 24, 2016 at 11:20 AM, Walter Underwood <wun...@wunderwood.org> > wrote: > >> Click through rate (CTR) is fundamental. That is easy to understand and >> integrates well with other business metrics like conversion. CTR is at >> least one click anywhere in the result set (first page, second page, …). >> Count multiple clicks as a single success. The metric is, “at least one >> click”. >> >> No hit rate is sort of useful, but you need to know which queries are >> getting no hits, so you can fix it. >> >> For latency metrics, look at 90th percentile or 95th percentile. Average >> is useless because response time is a one-sided distribution, so it will be >> thrown off by outliers. Percentiles have a direct customer satisfaction >> interpretation. 90% of searches were under one second, for example. Median >> response time should be very, very fast because of caching in Solr. During >> busy periods, our median response time is about 1.5 ms. >> >> Number of different queries per conversion is a good way to look how >> query assistance is working. Things like autosuggest, fuzzy, etc. >> >> About 10% of queries will be misspelled, so you do need to deal with that. >> >> Finding underperforming queries is trickier. I really need to write an >> article on that. >> >> “Search Analytics for Your Site” by Lou Rosenfeld is a good introduction. >> >> http://rosenfeldmedia.com/books/search-analytics-for-your-site/ < >> http://rosenfeldmedia.com/books/search-analytics-for-your-site/> >> >> Sea Urchin is doing some good work in search metrics: >> https://seaurchin.io/ <https://seaurchin.io/> >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> Search Guy, Chegg >> >> > On Feb 24, 2016, at 2:38 AM, Emir Arnautovic < >> emir.arnauto...@sematext.com> wrote: >> > >> > Hi Bill, >> > You can take a look at Sematext's search analytics ( >> https://sematext.com/search-analytics). It provides some of metrics you >> mentioned, plus some additional (top queries, CTR, click stats, paging >> stats etc.). In combination with Sematext's performance metrics ( >> https://sematext.com/spm) you can have full picture of your search >> infrastructure. >> > >> > Regards, >> > Emir >> > >> > -- >> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> > Solr & Elasticsearch Support * http://sematext.com/ >> > >> > >> > On 24.02.2016 04:07, William Bell wrote: >> >> How do others look at search metrics? >> >> >> >> 1. Search conversion? Do you look at searches and if the user does not >> >> click on a result, and reruns the search that would be a failure? >> >> >> >> 2. How to measure auto complete success metrics? >> >> >> >> 3. Facets/filters could be considered negative, since we did not find >> the >> >> results that the user wanted, and now they are filtering - who to >> measure? >> >> >> >> 4. One easy metric is searches with 0 results. We could auto expand >> the geo >> >> distance or ask the user "did you mean" ? >> >> >> >> 5. Another easy one would be tech performance: "time it takes in >> seconds to >> >> get a result". >> >> >> >> 6. How to measure fuzzy? How do you know you need more synonyms? How to >> >> measure? >> >> >> >> 7. How many searches it takes before the user clicks on a result? >> >> >> >> Other ideas? Is there a video or presentation on search metrics that >> would >> >> be useful? >> >> >> > >> >> > > > -- > *Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections > <http://opensourceconnections.com>, LLC | 240.476.9983 > Author: Relevant Search <http://manning.com/turnbull> > This e-mail and all contents, including attachments, is considered to be > Company Confidential unless explicitly stated otherwise, regardless > of whether attachments are marked as such. > -- *Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections <http://opensourceconnections.com>, LLC | 240.476.9983 Author: Relevant Search <http://manning.com/turnbull> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
