Hi Paras,
This is SO helpful, thank you. Quick question about your MRR metric -- do you
have binary human judgements for your suggestions? If no, how do you label
suggestions successful or not?
Best,
Audrey
On 2/24/20, 2:27 AM, "Paras Lehana" <paras.leh...@indiamart.com> wrote:
Hi Audrey,
I work for Auto-Suggest at IndiaMART. Although we don't use the Suggester
component, I think you need evaluation metrics for Auto-Suggest as a
business product and not specifically for Solr Suggester which is the
backend. We use edismax parser with EdgeNGrams Tokenization.
Every week, as the property owner, I report around 500 metrics. I would
like to mention a few of those:
1. MRR (Mean Reciprocal Rate): How high the user selection was among the
returned result. Ranges from 0 to 1, the higher the better.
2. APL (Average Prefix Length): Prefix is the query by user. Lesser the
better. This reports how less an average user has to type for getting the
intended suggestion.
3. Acceptance Rate or Selection: How many of the total searches are
being served from Auto-Suggest. We are around 50%.
4. Selection to Display Ratio: Did you make the user to click any of the
suggestions if they are displayed?
5. Response Time: How fast are you serving your average query.
The Selection and Response Time are our main KPIs. We track a lot about
Auto-Suggest usage on our platform which becomes apparent if you observe
the URL after clicking a suggestion on dir.indiamart.com. However, not
everything would benefit you. Do let me know for any related query or
explanation. Hope this helps. :)
On Fri, 14 Feb 2020 at 21:23, Audrey Lorberfeld - audrey.lorberf...@ibm.com
<audrey.lorberf...@ibm.com> wrote:
> Hi all,
>
> How do you all evaluate the success of your query autocomplete (i.e.
> suggester) component if you use it?
>
> We cannot use MRR for various reasons (I can go into them if you're
> interested), so we're thinking of using nDCG since we already use that for
> relevance eval of our system as a whole. I am also interested in the
metric
> "success at top-k," but I can't find any research papers that explicitly
> define "success" -- I am assuming it's a suggestion (or suggestions)
> labeled "relevant," but maybe it could also simply be the suggestion that
> receives a click from the user?
>
> Would love to hear from the hive mind!
>
> Best,
> Audrey
>
> --
>
>
>
--
--
Regards,
*Paras Lehana* [65871]
Development Engineer, *Auto-Suggest*,
IndiaMART InterMESH Ltd,
11th Floor, Tower 2, Assotech Business Cresterra,
Plot No. 22, Sector 135, Noida, Uttar Pradesh, India 201305
Mob.: +91-9560911996
Work: 0120-4056700 | Extn:
*11096*
--
*
*
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_IndiaMART_videos_578196442936091_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=_8ViuZIeSRdQjONA8yHWPZIBlhj291HU3JpNIx5a55M&m=CTfu2EkiAFh-Ra4cn3EL2GdkKLBhD754dBAoRYpr2uc&s=kwWlK4TbSM6iPH6DBIrwg3QCeHrY-83N5hm2HtQQsjc&e=
>
