[
https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219094#comment-14219094
]
Pedro Rodriguez commented on SPARK-1405:
----------------------------------------
I will take a look at those when I get a chance (after AMPCamp probably).
Here is a link to my GDrive with the NIPS data set:
https://drive.google.com/folderview?id=0B0zg9iArSQQaRE1OR1lLQWNjZG8&usp=sharing
The github repo here is my benchmarking/testing wrapper program around LDA in
case you want to see how I parse it in:
https://github.com/EntilZha/nips-lda-spark
Graphlab actually uses the same metric that I/we are using for LDA. If you
paste this into a tex editor it is easier to see:
\mathcal{L}( w | z) & = T * \left( \log\Gamma(W * \beta) - W *
\log\Gamma(\beta) \right) + \\
& \sum_{t} \left( \left(\sum_{w} \log\Gamma(N_{wt} + \beta)\right) -
\log\Gamma\left( W * \beta + \sum_{w} N_{wt} \right) \right) \\
& = T * \left( \log\Gamma(W * \beta) - W * \log\Gamma(\beta) \right) -
\sum_{t} \log\Gamma\left( W * \beta + N_{t} \right) + \\
& \sum_{w} \sum_{t} \log\Gamma(N_{wt} + \beta) \\
\\
\mathcal{L}(z) & = D * \left(\log\Gamma(T * \alpha) - T *
\log\Gamma(\alpha) \right) + \\
& \sum_{d} \left( \left(\sum_{t}\log\Gamma(N_{td} + \alpha)\right) -
\log\Gamma\left( T * \alpha + \sum_{t} N_{td} \right) \right) \\
\\
\mathcal{L}(w,z) & = \mathcal{L}(w | z) + \mathcal{L}(z)\\
N_{td} =\text{number of tokens with topic t in document d}\\
N_{wt} =\text{number of tokens with topic t for word w}
LDA (roughly) converges to -2.8e7 while graphlab (roughly) converges to -2.65e7
for comparable runtime (not iterations, couldn't figure out an easy way to get
num iterations ran oddly).
> parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib
> -----------------------------------------------------------------
>
> Key: SPARK-1405
> URL: https://issues.apache.org/jira/browse/SPARK-1405
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: Xusen Yin
> Assignee: Guoqiang Li
> Priority: Critical
> Labels: features
> Attachments: performance_comparison.png
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Latent Dirichlet Allocation (a.k.a. LDA) is a topic model which extracts
> topics from text corpus. Different with current machine learning algorithms
> in MLlib, instead of using optimization algorithms such as gradient desent,
> LDA uses expectation algorithms such as Gibbs sampling.
> In this PR, I prepare a LDA implementation based on Gibbs sampling, with a
> wholeTextFiles API (solved yet), a word segmentation (import from Lucene),
> and a Gibbs sampling core.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
