Greetings all, I have off and on been trying to make sense of topic models, and while doing a little searching I found a blog discussion at http://nlpers.blogspot.com/ which seemed quite relevant, and gave me a chance to ask a couple of questions that have been on my mind for a while...if you find topic models interesting at all, you should check out the original thread...
Anyway, if you aren't familiar with that blog, it is primarily by Hal Daume, but seems to have fairly good community participation as well, so seems like a good sounding board for quite a few issues of concern to the ACL community. Actually this blog has been mentioned in this group before, so if you haven't already checked it out you probably should. Just wanted to post this here to share my questions about topic models, in case you know the answers. :) ============================ Very nice thread, and in fact if you search for "evaluating topic models" on Google, it comes up as the number 1 hit (at least today, June 15, 2008). So how is it that I find myself posting to this blog? Well, one of the problems I think about quite a bit is how to identify terms from collections of documents that will describe the content of that collection and (maybe) discriminate that collection from the others. Given what topic models try to do, they seem at least like worthy candidates for the descriptive part of the above, or maybe even the discriminating too... Anyway, I've read a few topic model papers, and they usually go something like this - they start with an appealing introduction to the problem, that is we want to find terms that help us organize information, navigate through complex information spaces, etc. Then you hit this rather terrifying middle section of equations and terminology that usually takes up about half of the page limit (thus far I've mostly skipped those parts to be honest), and then there is an "evaluation" at the end of the paper that includes some tables that are essentially ranked lists of words that look pretty good. The ranked list of terms is indeed made up of terms that are descriptive, they are content rich words, and certainly they are useful pieces of information. But then, I think about that middle section of the paper, and I think about something like TF/IDF, which can do a fairly good job of some of the above with quite a bit less algorithmic and probably computational complexity. Or I think about measures of association like pointwise mutual information (PMI) that can be used to identify interesting bigrams that are often very descriptive of content, and I wonder if topic models are really providing anything that different or that much better than some of the techniques we already know. That's actually what I was looking for when I found this discussion - that is, has anyone of a similar frame of mind said "let's see how the lists of terms we get from a topic model compares to TF/IDF, etc."? Thus far I haven't found anything, but I'm still curious. To tie this back to the thread of evaluation - what about comparative evaluation? Most of the topic model work I've seen doesn't seem to go outside of the probabilistic box when looking for methods to compare to, so the discussion quickly focuses on different sorts of parameters and what not, but doesn't really seem to acknowledge that there might be completely different alternatives that accomplish the same task, and might do so pretty well. I'd actually be content to know if topic models provided something different than you would get from TF/IDF, etc, because as has been said evaluation of lists of ranked terms outside of some application setting is always a tricky business. (This is the same thing that has long bedeviled work on methods to identify and rank collocations, btw) But, at the very least it would be useful to know that the top 10 terms provided by a certain topic model correlates to X degree with the top 10 terms provided by TF-IDF, etc. Anyway, I fear this sounds painfully naive, but you seem like a friendly group. :) I think the truth is that when complicated methods like topic models are used to attack problems that look simple (give me the 10 words that best describe the content of this article or set of articles), there is a sort of natural tendency on the part of a person trying to solve that problem in the context of a larger problem to ask to whether all that machinery is really necessary or not, and that seems like it would best be settled to some degree via comparative evaluations with quite different methods that try to accomplish the same thing. Any guidance on any of this? Cordially, Ted -- Ted Pedersen http://www.d.umn.edu/~tpederse
