Hi Audrey, I work for Auto-Suggest at IndiaMART. Although we don't use the Suggester component, I think you need evaluation metrics for Auto-Suggest as a business product and not specifically for Solr Suggester which is the backend. We use edismax parser with EdgeNGrams Tokenization.
Every week, as the property owner, I report around 500 metrics. I would like to mention a few of those: 1. MRR (Mean Reciprocal Rate): How high the user selection was among the returned result. Ranges from 0 to 1, the higher the better. 2. APL (Average Prefix Length): Prefix is the query by user. Lesser the better. This reports how less an average user has to type for getting the intended suggestion. 3. Acceptance Rate or Selection: How many of the total searches are being served from Auto-Suggest. We are around 50%. 4. Selection to Display Ratio: Did you make the user to click any of the suggestions if they are displayed? 5. Response Time: How fast are you serving your average query. The Selection and Response Time are our main KPIs. We track a lot about Auto-Suggest usage on our platform which becomes apparent if you observe the URL after clicking a suggestion on dir.indiamart.com. However, not everything would benefit you. Do let me know for any related query or explanation. Hope this helps. :) On Fri, 14 Feb 2020 at 21:23, Audrey Lorberfeld - audrey.lorberf...@ibm.com <audrey.lorberf...@ibm.com> wrote: > Hi all, > > How do you all evaluate the success of your query autocomplete (i.e. > suggester) component if you use it? > > We cannot use MRR for various reasons (I can go into them if you're > interested), so we're thinking of using nDCG since we already use that for > relevance eval of our system as a whole. I am also interested in the metric > "success at top-k," but I can't find any research papers that explicitly > define "success" -- I am assuming it's a suggestion (or suggestions) > labeled "relevant," but maybe it could also simply be the suggestion that > receives a click from the user? > > Would love to hear from the hive mind! > > Best, > Audrey > > -- > > > -- -- Regards, *Paras Lehana* [65871] Development Engineer, *Auto-Suggest*, IndiaMART InterMESH Ltd, 11th Floor, Tower 2, Assotech Business Cresterra, Plot No. 22, Sector 135, Noida, Uttar Pradesh, India 201305 Mob.: +91-9560911996 Work: 0120-4056700 | Extn: *11096* -- * * <https://www.facebook.com/IndiaMART/videos/578196442936091/>
