Hi Eric,

>         Thanks Olga, BUT I don't see how Spearman's (or other) rank
>         correlation formula measures overlap between 2 ranked lists
>         containing some different words.

in a recent work, I had a similar evaluation problem, yet not exactly the
same. I had gold standard rankings of locations given an input object
(crowdsourced asking the question "where is object X it likely ti be
found?"). I then had a system produce similar rankings on the same set of
locations given an object. In theory the sets of locations are the same,
but in practice the system doesn't have perfect recall, so its list is
shorter, the overlap is not total between the two lists. In my case, I
"solved" this problem by adding the missing items at the bottom of the
ranking. This could be a quick fix for your problem too: just add the items
of list A that are missing from list B at the bottom of the ranking for A
and vice versa. not really elegant, admittedly.

The problem with a ranking correlation metric like Kendall's Tau, in my
case, is that I needed to give more importance to guessing correctly the
top of then list, and less importance to the bottom. My understanding is
that your evaluation should have this feature too. I ended up using nDCG (
https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG). I
found this metric accurate, in that it reflects my intuitions on my data,
but not very intuitive to interpret. So, in that work, I also measured
precision at 1 and at 3, that is, the number of time my system guessed the
first items and the top 3 items correctly, respectively.
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list

Reply via email to