Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/7064#issuecomment-117804916
topicDistributions does not include the empty documents because those
vertices are dropped during learning. Look at this line:
[https://github.com/apache/spark/blob/1ce6428907b4ddcf52dbf0c86196d82ab7392442/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala#L186]
I'd propose 1 of these 2 options:
(1) We simplify things by simply making sure all empty documents are
dropped before learning (tested in a unit test). We should also add
documentation in the Scala doc to make it very clear. The downside is that
users who want to join the topicDistributions with the original data will need
to filter the original data---but I feel like they should anyways.
(2) Alternatively, we could add the empty document vertices in at the end
when creating the DistributedLDAModel. This might be easier than modifying
learning itself.
I'd prefer option (1). What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
