Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/7801#discussion_r36006448
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/clustering/LDASuite.scala ---
@@ -274,32 +280,13 @@ class LDASuite extends SparkFunSuite with
MLlibTestSparkContext {
> -3.69051285096
*/
- assert(ldaModel.logPerplexity(docs) ~== -3.690D relTol 1E-3D)
+ // Gensim's definition of perplexity is negative our (and Stanford
NLP's) definition
--- End diff --
Did you check Stanford NLP? I thought our definition matches Stanford NLP.
I'm having trouble finding clear documentation online, but I found 2
indications that it matches ours:
* [http://nlp.stanford.edu/software/tmt/tmt-0.2/] "searches for the
parameters that minimize the model's perplexity"
*
[https://github.com/stanfordnlp/CoreNLP/blob/6d98bc6aa3f390eeeaa8659088445aad60d87ae1/src/edu/stanford/nlp/classify/NBLinearClassifierFactory.java#L175]:
uses negative log likelihood
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
