Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/7307#discussion_r34305062
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala
---
@@ -77,37 +89,47 @@ class LDA private (
* Concentration parameter (commonly named "alpha") for the prior placed
on documents'
* distributions over topics ("theta").
*
- * This is the parameter to a symmetric Dirichlet distribution.
+ * This is the parameter to a Dirichlet distribution.
*/
- def getDocConcentration: Double = this.docConcentration
+ def getDocConcentration: Vector = this.docConcentration
/**
* Concentration parameter (commonly named "alpha") for the prior placed
on documents'
* distributions over topics ("theta").
*
- * This is the parameter to a symmetric Dirichlet distribution, where
larger values
- * mean more smoothing (more regularization).
+ * This is the parameter to a Dirichlet distribution, where larger
values mean more smoothing
+ * (more regularization).
*
- * If set to -1, then docConcentration is set automatically.
- * (default = -1 = automatic)
+ * If set to a vector of -1, then docConcentration is set automatically.
+ * (default = a vector of -1 = automatic)
*
* Optimizer-specific parameter settings:
* - EM
+ * - Currently only supports symmetric distributions, so values in
the vector must be the same
* - Value should be > 1.0
* - default = (50 / k) + 1, where 50/k is common in LDA libraries
and +1 follows
* Asuncion et al. (2009), who recommend a +1 adjustment for EM.
* - Online
- * - Value should be >= 0
- * - default = (1.0 / k), following the implementation from
+ * - Values should be >= 0
+ * - default = uniformly (1.0 / k), following the implementation from
* [[https://github.com/Blei-Lab/onlineldavb]].
*/
- def setDocConcentration(docConcentration: Double): this.type = {
+ def setDocConcentration(docConcentration: Vector): this.type = {
this.docConcentration = docConcentration
this
}
+ /** Replicates Double to create a symmetric prior */
+ def setDocConcentration(docConcentration: Double): this.type = {
--- End diff --
What if the user sets this and then sets K? Maybe we should have a private
var docConcentrationScalar which tracks whether it's been set via a scalar (so
that setK can update the Vector if needed). That scalar could be unset if the
user calls the other setter (with a Vector).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
