Denis Turdakov created SPARK-2199:
-------------------------------------
Summary: Distributed probabilistic latent semantic analysis in
MLlib
Key: SPARK-2199
URL: https://issues.apache.org/jira/browse/SPARK-2199
Project: Spark
Issue Type: Improvement
Components: MLlib
Affects Versions: 1.1.0
Reporter: Denis Turdakov
Probabilistic latent semantic analysis (PLSA) is a topic model which extracts
topics from text corpus. PLSA was historically a predecessor of LDA. However
recent research shows that modifications of PLSA sometimes performs better then
LDA[1]. Furthermore, the most recent paper by same authors shows that there is
a clear way to extend PLSA to LDA and beyond[2].
We should implement distributed versions of PLSA. In addition it should be
possible to easily add user defined regularizers or combination of them. We
will implement regularizers that allows
• extract sparse topics
• extract human interpretable topics
• perform semi-supervised training
• sort out non-topic specific terms.
[1] Potapenko, K. Vorontsov. 2013. Robust PLSA performs better than LDA. In
Proceedings of ECIR'13.
[2] Vorontsov, Potapenko. Tutorial on Probabilistic Topic Modeling: Additive
Regularization for Stochastic Matrix Factorization.
http://www.machinelearning.ru/wiki/images/1/1f/Voron14aist.pdf
--
This message was sent by Atlassian JIRA
(v6.2#6252)
