mmmm, I have a question about that. If you use the tf*idf vectors insteed of tf vectors to feed the CVB0 LDA, you are going to generate a matrix as big as the tf vectors do, but the terms smoothed with the idf measure.
I think that if you want to reduce the size of the matrix you have to use the maxDFPercent parameter of the TFIDFConverter.processTfIdf method (or in the DictionaryVectorizer.createTermFrequencyVectors method). Is it right? On Tue, May 29, 2012 at 11:12 PM, Jake Mannix <[email protected]> wrote: > > > On Tue, May 29, 2012 at 5:03 PM, Runkel, Timothy J < > [email protected]> wrote: > > > > Now, given the topic by terms association vectors, how can we take a new > > document (in Term Frequency format using the same dictionary as the > trained > > documents and ignoring any few terms not found) and query the model to > rank > > its top topic matches? Academic papers seem to gloss over this task. > > > > Yeah, this is totally straightforward, but you're right, it's not usually > described well. Note, although LDA doesn't technically follow the correct > "theory" if you use TF-IDF vectors, in "practice" you can often get better > results this way (ie. train with it, and then do the final topic -> > document association step with this as well), in that you get less of the > crappy super-common terms at the top of all of your topics. > > > > -- > > -jake >
