[
https://issues.apache.org/jira/browse/MAHOUT-458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025792#comment-13025792
]
Jake Mannix commented on MAHOUT-458:
------------------------------------
Per the thread on the mailing list "Determining Document Cluster Probabilities
with LDA", we should not have closed this bug. I don't know why it ever got
closed, but it's a glaring deficiency in the LDA impl we have. I have written
a patch for this on my github branch (lda_gamma_output), which does a couple of
things: a) it computes the document/topic distribution, b) it is gracefully
restartable, using the current on-hdfs state of the iteration, and c) has a
sequential version of the algorithm, for non-hadoop use. c) isn't quite
working yet.
> The LDA output does not include the topic-probability distribution per
> document (p(z|d)). It outputs only the topics and corresponding words.
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-458
> URL: https://issues.apache.org/jira/browse/MAHOUT-458
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.4
> Reporter: Himanshu Gahlot
> Assignee: Jeff Eastman
> Fix For: 0.4
>
> Attachments: MAHOUT-458.patch
>
>
> The current implementation of LDA outputs only topics and their words. Many
> applications need the p(z|d) values of a document to use this vector as a
> reduced representation of the document (dimensionality reduction of
> document). We need to introduce a new key which would keep track of the gamma
> values for each document (as obtained from the document.infer() method) and
> writes these to the output stream and finally, PrintLDATopics should output
> these values per document id. Also, outputting the probabilities of words in
> a topic would also provide a more meaningful output.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
