That is a nice summary. A couple of thoughts to add for those who aren't information-retrieval nerds (this helped solidify TF and IDF in my head >20 years ago):
* I like to think of TF (whatever it's formulation) as how *characteristic *a term is for a document. "dog" occurs once, then this article isn't really about "dog". * Similarly, IDF is how *distinctive* a term is for a document. "the" shows up a lot in English text—so it is characteristic—but it shows up in every text, so it isn't distinctive. So, in a corpus of English documents, "the" is characteristic (high TF) but not distinctive (low IDF) for any given document. OTOH, in a corpus that's 99% Swahili documents, and a small handful of English documents, "the" is both characteristic (because the docs are in English) and distinctive (because most other docs are not). Thus, TF is about a given document, and IDF is about the corpus it is part of. * Those cool graphs in that blog post? Those were obviously done with Desmos ( https://www.desmos.com ) a powerful free HTML5 graphing calculator. We use it to look at scoring components to understand how different formulas behave, and how modifying various parameters affects them. I use it so much, I just wanted to share it. —Trey Trey Jones Software Engineer, Discovery Wikimedia Foundation On Fri, Mar 18, 2016 at 7:36 AM, Guillaume Lederrey <[email protected] > wrote: > Yep, nice reading! > > On Fri, Mar 18, 2016 at 12:28 AM, Dan Garry <[email protected]> wrote: > > This was a really helpful read for me! Thanks for sending. :-) > > > > Dan > > > > On 14 March 2016 at 16:21, Tomasz Finc <[email protected]> wrote: > >> > >> Here is a good write up that breaks it down > >> > >> > >> > http://opensourceconnections.com/blog/2015/10/16/bm25-the-next-generation-of-lucene-relevation/ > >> > >> Given the recent threads about exploring BM25, i thought this was a > >> good introduction to the difference between the two. > >> > >> Cheers. > >> > >> --tomasz > >> > > >
_______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
