On Sunday May 2 I attended two tutorials. The first was "Statistical Language Models for Information Retrieval" by ChengXiang Zhai (http://www-faculty.cs.uiuc.edu/~czhai/)
The basic idea was to turn information retrieval (IR) on it's head. :) IR is in general concerned with providing a few search terms to a collection of documents, and finding those documents that are most relevant to those terms. It is certainly reasonable to think of web searching as IR, where you issue search terms and the documents you get back are web pages. I'll use the more classical terminology of document and query, but you can think search term and web page if you like! In classical IR, the common technique is to find those documents that match a users query. For example, I might use the terms "duluth skateboard", and I would find the documents in the collection that contain the words "duluth" and "skateboard". This is fine, but it is a document-centered view of things. There will be a certain set of queries that will be optimal for given documents, and it is up to the user to figure out what those queries would/should be. This classical view of IR does not take into account the context of the search, or the nature of the user. For example, consider the term "IR". For folks like us, IR probably nearly always means "information retrieval". However, for many other kinds of users IR might mean "infrared". There is no way for the keyword based approach to Information Retrieval described above to "know" that I mean IR as Information Retrieval. So, the idea behind the approach described in the tutorial is to use a users query history to rank documents. In effect, we learn a model that tells us how likely that model is to have generated a particular query. (I'm still trying to figure this idea out completely, btw!). So, when a user makes a query and then follows a link to a particular document, that query is recorded as a successful representation or model of that particular document, for that user. Typically IR systems (image a web search engine) return a ranked list of documents, so if a user makes a query for a particular set of terms, and then scrolls down through the output until they find a document that seems relevant to them (let's say at rank 35), then all of a sudden that document that was ranked 35 now becomes strongly associated with that query. Imagine that I issue a query for "duluth parks" and all I get back are documents about Duluth Car Parks. However, maybe down at position 30 there is a document about Duluth Nature Parks, which is what I "really" meant. For me then, the query "duluth parks" becomes stronly associated with documents that contain terms like "duluth nature parks", which reflects the fact that I view "parks" and "nature parks" as synonyms (but maybe not everyone would do that). I won't get into the details of the language model, but fortunately it's quite simple (a basic unigram model with various sorts of smoothing). The slides for the tutorial and additional references can be found at: http://sifaka.cs.uiuc.edu/lmir Hope this made a little sense at least. Let me know if you find this interesting or you have any additional questions or comments! Ted -- Ted Pedersen http://www.d.umn.edu/~tpederse ------------------------ Yahoo! Groups Sponsor ---------------------~--> Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar. Now with Pop-Up Blocker. Get it for free! http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/x3XolB/TM ---------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/nlpatumd/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
